A  SPACE  ACQUISITION  LEADING  INDICATOR 
BASED  ON  SYSTEM  INTEROPERATION  MATURITY 

THESIS 


Jason  T.  Shibata,  Major,  USAF 

AFIT/GSE/ENV/ 1 0-D02DL 

DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 

Wright-Patterson  Air  Force  Base,  Ohio 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 


The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official 
policy  or  position  of  the  United  States  Air  Force,  Department  of  Defense,  or  the  U.S. 
Government. 


AFIT/GSE/ENV/ 1 0-D02DL 


A  SPACE  ACQUISITION  LEADING  INDICATOR 
BASED  ON  SYSTEM  INTEROPERATION  MATURITY 

THESIS 

Presented  to  the  Faculty 

Department  of  Systems  and  Engineering  Management 
Graduate  School  of  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Air  University 

Air  Education  and  Training  Command 
In  Partial  Fulfillment  of  the  Requirements  for  the 
Degree  of  Master  of  Science  in  Space  Systems  Engineering 

Jason  T.  Shibata,  BS 
Major,  USAF 

December  2010 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 


AFIT/GSE/ENV/ 1 0-D02DL 


A  SPACE  ACQUISITION  LEADING  INDICATOR 
BASED  ON  SYSTEM  INTEROPERATION  MATURITY 


Jason  T.  Shibata,  BS 
Major,  USAF 


Approved: 


//SIGNED// 

December  2010 

John  M.  Colombi,  Ph.D.  (Chainnan) 

Date 

//SIGNED// 

December  2010 

David  R.  Jacques,  Ph.D.  (Member) 

Date 

//SIGNED// 

December  2010 

Joseph  R.  Wirthlin,  Lt  Col,  USAF  (Member) 

Date 

AFIT/GSE/ENV/ 1 0-D02DL 


Abstract 

The  Department  of  Defense‘s  space  acquisition  enterprise  has  experienced 
numerous  challenges  since  the  advent  of  space  power.  Space  borne  capabilities  are 
needed  now  more  than  ever,  but  space  acquisition  programs  frequently  fail  to  meet  cost 
and  schedule  goals.  The  decades  of  space  acquisition  experience  form  a  rich  history  that 
can  be  used  to  build  a  leading  indicator  of  program  success  and  serve  as  an  enabler 
toward  effective  program  execution.  First,  the  space  acquisition  areas  of  greatest  concern 
were  determined  to  be  cost,  schedule  and  requirements.  These  areas  can  be  considered  as 
systems  that  are  composed  of  the  people,  processes  and  products  that  work  together  to 
execute  the  program.  Their  effective  interoperation  is  vital  toward  achieving  program 
success.  Second,  the  vital  interoperation  characteristics,  or  attributes  that  each  system 
must  possess  to  be  successful,  can  be  extracted  from  past  space  acquisition  lessons 
learned  and  placed  into  an  interoperability  maturity  model.  The  maturity  model  can  then 
be  used  to  capture  the  relative  maturity  of  the  progranTs  major  systems  and  their  ability 
to  interoperate  within  the  context  of  each  critical  characteristic.  Third,  the  maturity 
model  forms  the  basis  for  an  interoperability  measurement  using  the  method  developed 
by  Dr.  Thomas  Ford,  where  higher  levels  of  interoperability  maturity  will  result  in  a 
higher  interoperability  score.  Finally,  this  process  is  demonstrated  with  three  recent 
space  programs.  This  application  demonstrates  how  the  interoperability  score  can  be 
used  as  a  leading  indicator  with  interpretive  analysis  provided. 
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A  SPACE  ACQUISITION  LEADING  INDICATOR 
BASED  ON  SYSTEM  INTEROPERATION  MATURITY 

I.  Introduction 

The  U.S.  Army  is  one  of  the  largest  users  of  space-based  capabilities  in  DOD.  As  the 
Army  transforms,  its  operational  characteristics  will,  in  large  part,  be  achieved  through 
the  use  and  exploitation  of  transformational  space  systems.  This  dependency  requires  the 
Army  to  actively  participate  in  defining  space  related  capability  needs  that  ensure 
necessary  force  structure  and  systems  are  developed  and  acquired  to  enable  the  land 

force  to  conduct  the  full  range  of  military  operations  now  and  in  the  future. 

— Army  Space  Policy  (Headquarters,  Department  of  the  Anny,  2009) 

Background 

There  is  little  doubt  that  the  tenets  of  network  centric  warfare  provide  military 
forces  a  decided  advantage  on  the  battlefield.  Robustly  networked  forces  share 
information  and  collaborate  resulting  in  synchronized  battlespace  effects,  greater  speed  of 
command,  and  increased  lethality,  survivability  and  responsiveness  (Department  of 
Defense,  2001).  The  ability  to  connect  and  interoperate  between  people  and  forces  is  a 
necessary  condition  toward  enabling  mission  effectiveness  in  virtually  any  collaborative 
environment,  including  acquisition. 

Interoperability  has  grown  within  the  defense  community  from  a  buzzword  to  a 
mandatory  system  requirement.  Despite  its  importance,  the  study  of  interoperability 
measurement  has  been  disparate  and  remains  largely  unproven.  Most  interoperability 
measurement  methods  focus  on  qualitative  means  rather  than  quantitative,  and  no  one 
method  has  been  accepted  as  the  de  facto  standard.  In  2007,  there  were  over  30  distinct 
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definitions,  over  60  types  and  at  least  14  interoperability  measurement  models  in 
existence  (Ford,  Colombi,  Graham,  &  Jacques,  2007). 

The  Department  of  Defense ‘s  (DOD)  approach  to  interoperability  is  manifested  in 
the  Net-Ready  Key  Performance  Parameter  (NR-KPP)  and  requires  that  joint  systems 
adhere  to  compliant  solution  architectures,  the  Net-Centric  Data  and  Services  Strategy, 
technical  standards  and  interfaces  through  the  Global  Information  Grid  (GIG)  Technical 
Guidance,  DOD  Information  Assurance  (IA)  requirements,  and  DOD  supportability 
requirements  (CJCSI,  2008).  The  NR-KPP  provides  a  framework  and  data  strategy  to 
enable  interoperability,  but  does  not  specify  how  to  measure  interoperability  nor  does  it 
establish  interoperability  performance  standards. 

Similarly,  the  DOD‘s  space  acquisition  community  does  not  have  a  way  to 
quantifiably  measure  the  effectiveness  of  its  acquisition  programs1  interoperations.  It 
generally  uses  a  series  of  gates  and  reviews  that  demand  various  levels  of  program 
maturity  and  rigor.  These  gates  and  reviews  enforce  good  acquisition  discipline  and,  in 
theory,  provide  the  program  with  the  best  chance  to  deliver  capability  to  the  warfighter  in 
a  time  confident  manner.  Although  the  term  interoperability  may  seem  foreign  when 
viewed  from  within  the  acquisition  process,  the  major  forces  that  affect  the  outcome  of  an 
acquisition  program;  organizations/people,  processes,  infonnation  and  systems,  must 
indeed  be  interoperable  with  one  another  and  aligned  toward  a  common  purpose. 

Major  Thomas  Ford‘s  seminal  work  supplied  the  first  quantifiable  method  for 
interoperability  measurement  (Ford,  2008),  and  fonned  the  initial  links  between 
interoperability  measurement  and  operational  mission  effectiveness.  He  introduced  the 
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concept  of  -eonfrontational  interoperability”  where  interoperability  is  measured  between 
two  opposing  systems,  and  where  one  system‘s  ability  to  control  the  other  results  in  a 
higher  degree  of  operational  mission  effectiveness. 

This  thesis  is  founded  on  Ford‘s  inaugural  work,  and  extends  his  ideas  further  into 
the  realm  of  collaborative  interoperability  within  the  context  of  space  acquisition.  As 
implied  by  the  opening  quote,  there  is  a  distinct  tie  between  the  ability  to  interoperate  and 
operational  mission  effectiveness.  This  paper  asserts  that  the  same  relationships  exist 
within  an  acquisition  program,  where  greater  acquisition  interoperability  results  in  an 
increased  ability  to  meet  requirements  on  time  and  on  schedule.  A  method  to  measure 
space  acquisition  interoperability  is  presented  and  provides  the  initial  building  blocks 
toward  a  quantifiable  means  to  assess  and  even  predict  space  acquisition  perfonnance. 

Hypotheses/Research  Objectives 

The  lessons  learned  from  past  space  acquisition  failures  can  be  used  to  detennine 
how  a  program1  s  cost,  schedule  and  requirements  interoperate  to  produce  a  particular 
outcome.  These  relationships  and  characteristics  can  be  extracted,  placed  into  a  maturity 
model,  and  then  evaluated  using  Ford‘s  interoperability  measurement  method  to  create  an 
indicator  of  effective  acquisition  interoperation.  This  measurement  method  can  then  be 
applied  to  current  and  future  acquisition  programs  to  indicate  or  predict  to  what  degree 
the  components  of  a  program  interoperate.  The  measure  incorporates  the  benefit  of 
hindsight  to  detennine  whether  or  not  the  program  is  likely  to  achieve  its  requirements  on 
time  and  on  schedule  based  upon  its  individual  components4  ability  to  interoperate. 
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Methodology 

The  research  method  first  examines  past  space  acquisition  failures  for  key  factors 
or  characters  that  dominated  the  progranTs  eventual  outcome.  These  failures  can  then  be 
used  to  compose  an  acquisition  maturity  model  founded  upon  earlier  works  such  as  the 
Levels  of  Information  Systems  Interoperability  (LISI)  model  and  the  Organizational 
Interoperability  Maturity  Model  (OIM). 

Finally,  a  measure  of  acquisition  effectiveness  will  be  developed  and  applied 
using  Ford‘s  work  and  the  aforementioned  acquisition  maturity  model. 

Assumptions/Limitations 

The  intent  of  this  work  is  to  detennine  the  crucial  acquisition  characteristics  that 
should  be  specifically  considered  when  measuring  acquisition  interoperability.  These 
characteristics  will  be  developed  based  on  past  lessons  learned,  and  will  be  subjective  in 
nature.  The  measurement  will  reward  a  greater  maturity  level  in  a  key  characteristic,  but 
cannot  explain  exactly  how  much  was  gained  because  of  it.  For  instance,  an  acquisition 
failure  may  be  caused  by  funding  instability,  but  no  quantifiable  method  exists  to 
detennine  exactly  how  much  the  schedule,  cost  and  perfonnance  were  impacted. 

Additionally,  Ford‘s  work  described  how  the  need  for  interoperability  varies  with 
time.  A  thorough  examination  of  the  time-dependencies  of  acquisition  interoperability  is 
not  presented  here  but  is  a  good  candidate  for  future  research. 

Lastly,  this  thesis  will  only  examine  interoperability  within  the  specific  context  of 
space  acquisition.  It  is  likely  that  the  ideas  presented  in  this  paper  are  relevant  to  other 
contexts;  however  those  aspects  will  not  be  addressed  here.  It  is  believed  that  these  vital 


4 


interoperability  aspects  at  their  core  are  universal  and  are  necessary  factors  to  enable 
collaborative  interoperability  between  systems  and  forces.  Therefore,  the  lessons  learned 
from  the  space  acquisition  domain  could  serve  as  relevant  factors  for  interoperability  in 
any  domain  at  their  highest  level  of  extraction. 

Implications 

The  space  acquisition  maturity  model  and  acquisition  interoperability 
measurement  supply  an  initial  approach  to  quantify  whether  or  not  a  space  program  will 
be  effective.  In  other  words,  the  measurement  can  help  predict  how  likely  the  program 
will  succeed  in  accomplishing  the  required  level  of  perfonnance  within  cost  and  schedule 
constraints  -  a  leading  indicator.  This  method  could  be  used  to  evaluate  a  program1  s 
maturity  and  progress  throughout  its  lifecycle,  and  potentially  flag  program  risk  areas 
before  they  are  realized. 

Preview 

The  literature  review  will  provide  background  essential  to  understanding  Ford‘s 
interoperability  measurement  method,  interoperability  maturity  models,  and  how  the 
models  can  be  used  to  facilitate  interoperability  measurements.  An  examination  of  the 
key  areas  (systems)  involved  in  and  driving  causes  of  space  acquisition  failures  follows. 

The  methodology  section  fuses  the  concepts  and  infonnation  from  the  literature 
review  to  demonstrate  a  method  to  measure  space  acquisition  interoperability.  An 
acquisition  interoperability  maturity  model  will  be  built  based  upon  the  key  and  driving 
causes  of  space  acquisition  failures. 
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In  the  analysis  and  results  chapter,  the  space  acquisition  maturity  model  will  be 
used  to  measure  the  interoperability  of  several  space  programs  by  means  of  Ford‘s 
interoperability  measurement  method  and  will  discuss  the  use  of  the  measurements  as  a 
leading  indicator  of  space  acquisition  effectiveness. 

The  conclusions  and  recommendations  section  will  examine  the  results  of  the 
aforementioned  interoperability  measure  as  a  leading  indicator  and  will  discuss  the  utility 
of  a  leading  indicator  with  respect  to  ongoing  and  future  space  programs.  Suggestions 
for  future  research  and  maturation  of  the  leading  indicator  will  follow. 
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II.  Literature  Review 


“...a  process  and  procedure  for  establishing  goals  for  improving  the 
efficiency  and  effectiveness  of  government  agencies  operations  and  the 
ability  to  deliver  goods  and  sendees  to  the  public  using  Information 
Technology >.  The  goals  must  be  measurable.  ” 

— Clinger  Cohen  Act,  Public  Law  104-106 

Chapter  Overview 

The  purpose  of  this  chapter  is  to  provide  a  summary  of  Ford‘s  method, 
interoperability  maturity  models,  and  past  space  acquisition  failures  and  their  root  causes. 
This  literature  review  lays  the  foundation  for  the  development  of  an  acquisition 
effectiveness  maturity  model  and  later,  a  quantifiable  method  for  measuring  space 
acquisition  interoperability.  A  review  of  leading  indicators  for  DOD  acquisition 
programs  will  also  be  discussed  to  validate  the  utility  of  the  interoperability 
measurement. 

The  literature  review  first  provides  a  brief  summary  of  the  foundational  works 
that  this  paper  relies  and  builds  upon.  The  basic  tenets  of  Ford‘s  interoperability 
measurement  method  and  the  interoperability  maturity  models  are  essential  knowledge  to 
understand  the  concepts  and  applications  presented  in  this  thesis.  A  summary  of  past 
space  acquisition  failures  follows.  Various  sources  were  used  to  identify  the  root  causes 
of  the  space  acquisition  failures  of  the  past,  and  these  root  causes  will  later  serve  as  chief 
contributors  to  an  acquisition  interoperability  maturity  model  in  the  methodology  section 
of  this  paper. 
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Before  beginning,  it  is  critically  important  that  a  standard  definition  of  the  term 
-interoperability”  is  provided.  Ford‘s  paper,  the  2007  survey  on  interoperability 
measurement,  LISI  and  OIM  chose  to  use  the  DOD‘s  original  definition  from  Joint 
Publication  1-02  (Department  of  Defense,  2005)  as  the  standard.  It  states  that 
interoperability  is  — [liability  of  systems,  units,  or  forces  to  provide  services  to  and 
accept  services  from  other  systems,  units,  or  forces  and  to  use  the  services  so  exchanged 
to  enable  them  to  operate  effectively  together”  (Ford,  Colombi,  Graham,  &  Jacques, 
2007).  This  thesis  will  remain  consistent  with  this  definition. 

Ford’s  Interoperability  Measurement  Method 

Ford  created  a  method  to  measure  confrontational  interoperability  based  on  and 
constrained  by  an  operational  scenario  and  the  systems  and  processes  that  execute  the 
activities  contained  in  the  scenario.  Ford’s  basic  process  to  define  the  interoperability 
measurement  can  be  described  by  the  figure  below,  taken  from  Ford’s  earlier  work: 
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Figure  1.  Interoperability  Measurement  Process  (Ford,  2008) 

The  first  step  in  Ford‘s  interoperability  measurement  process  is  to  define  the 
purpose  of  the  measurement.  Defining  the  purpose  is  necessary  to  adequately  scope  the 
interoperability  measurement,  and  serves  as  the  anchor  for  the  interoperability  analysis. 
Once  the  purpose  has  been  determined,  the  process  that  serves  or  is  the  subject  of  the 
purpose  must  be  developed.  In  Ford‘s  method  and  this  paper,  operational  scenarios 
defined  the  process  for  analysis.  In  this  paper,  acquisition  programs  will  provide  the 
processes  for  analysis. 

According  to  Defense  Acquisition  University,  Measures  of  Effectiveness  (MOE) 
are  -a- measure  of  operational  success  that  must  be  closely  related  to  the  objective  of  the 
mission  or  operation  being  evaluated.  For  example,  the  number  of  enemy  submarines 
sunk  or  enemy  tanks  destroyed  may  be  satisfactory  MOEs  if  the  objective  is  to  destroy 
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such  weapons  systems... A  meaningful  MOE  must  be  quantifiable  and  a  measure  to  what 
degree  the  real  objective  is  achieved.”  Ford‘s  work  was  chiefly  focused  on 
confrontational  interoperability  which  used  MOEs  based  on  the  effectiveness  of  friendly, 
or  -blue”  forces  compared  against  the  effectiveness  of  enemy,  or  -red”  forces  (e.g. 
number  of  enemy  submarines  sunk).  Collaborative  interoperability  on  the  other  hand  will 
use  MOEs  based  on  the  effectiveness  of  friendly  systems  to  operate  with  each  other  (e.g. 
number  of  successful  communications  messages  sent  and/or  received). 

Many  definitions  of  the  word  -system”  exist,  but  this  research  will  maintain 
Ford‘s  definition  of  a  system  as,  — a  entity  comprised  of  related  interacting  elements, 
which  act  together  to  achieve  a  purpose”  and  is  -broad  enough  to  include  a  wide  variety 
of  systems  including,  but  not  limited  to,  technical,  biological,  environmental, 
organizational,  conceptual,  physical,  and  philosophical,  among  others.”  (Ford,  2008). 

This  broad  view  is  critical  to  developing  a  flexible  method  to  measure  interoperability 
and  can  be  applied  to  the  space  acquisition  domain.  The  notation  for  a  set  of  systems  is  S 
=  {si,  S2...s„ }  where  S  represents  the  complete  set  of  systems  participating  in  the 
operational  scenario,  and  s„  represents  the  individual  systems. 

Once  the  operational  scenario  and  systems  have  been  chosen,  the  interoperability 
characters  must  be  defined.  At  a  high  level,  characters  describe  salient  and  distinct 
attributes  of  a  system  (e.g.  size,  shape,  function,  etc.).  The  notation  for  characters  is  X  = 
{xi,  X2...Xn }  where  X  represents  the  set  of  characters  used  to  model  the  systems,  and  xn 
represents  the  characters  used  to  describe  the  individual  systems.  The  states  of  these 
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characters  (e.g.  the  character  -size”  could  have  a  state  of  -2-5  ft”),  is  noted  as  C  =  {ci, 
C2...cn}  and  follows  Ford‘s  formal  definition: 

DEFINITION  (System  Characterization):  Given  a  set  of  systems  S,  then  X :  S 
C  is  a  function  which  maps  systems  to  a  set  of  character  states  C  and  X  is  called 
the  characterization  of  S.  (Ford,  2008) 

For  interoperability  measurement,  only  certain  types  of  characters,  known  as 
interoperability  characters,  are  used.  These  characters  describe  how  the  systems  in  the 
scenario  interoperate,  and  are  generally  based  on  the  actions  the  systems  must  perfonn  to 
or  accept  from  each  other.  These  interactions,  much  like  a  conversation  between  two 
people,  have  two  primary  components.  One  party  transmits  the  action,  the  other  receives 
it.  Ford‘s  work  provides  a  table  of  potential  interoperability  characters: 


Table  1.  Interoperability  Characters  (Ford,  2008) 


Interoperability  Pairs 

Interoperability  Type 

Character 

Provide  O  Accept 

General 

Interoperate 

Transmit  O  Receive 

Communication 

Communicate 

Attack  O  Attacked 

Confrontational 

Attack 

Impact  O  Impacted 

Confrontational 

Impact 

Detect  o  Detected 

Technological 

Detect 

Publish  O  Subscribe 

Net-Centric 

Service 

Occupy  O  Accommodate 

Spatial 

Accommodate 

Serve  O  Be  Served 

Human 

Service 

Give  O  Take 

Human 

Share 

Buy  O  Sell 

Business 

Trade 

Pay  O  Get  Paid 

Financial 

Transact 

Output  O  Input 

Traditional  System 

Outputlnput 

Lead  O  Follow 

Organizational 

Dance 

Order  O  Obey 

Human.  Organizational 

Command 

Produce  o  Consume 

Business.  Human 

Economy 

Transport  O  Transported  by 

Business 

Transport 

In  Ford‘s  method,  the  states  of  these  interoperability  characters  are  usually 
denoted  using  absence  or  presence  states.  That  is,  C  =  {0,1}  where  -zero”  indicates  the 
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absence  and  -one”  indicates  the  presence  of  a  specific  interoperability  character.  Ford‘s 
interoperability  characters  also  capture  the  direction  of  the  interoperation.  A 
conversation  between  two  people  is  directional  in  nature.  One  person  is  transmitting 
(speaking)  a  message  in  the  direction  of  the  other  person,  who  is  receiving  (listening)  the 
transmitted  message.  A  conversation  between  two  people  could  be  described  as  a  -bi¬ 
directional  interoperation”  because  both  parties  are  able  to  transmit  and  receive  messages 
If  one  person  was  mute  or  deaf,  the  interoperability  between  the  conversational  parties 
could  be  described  as  a  -uni-directional  interoperation”  because  only  one  party  is  able  to 


Figure  2,  Directional  Interoperability  (Ford,  2008) 

Once  the  systems  and  their  characters  are  defined  for  a  specific  operational 
scenario  and  interoperability  measurement  purpose,  they  are  instantiated”  or  listed  as  a 
sequence  of  states  for  each  system.  For  example,  an  aircraft  could  have  characters  of 
-type”,  -load”  and  -speed”,  and  could  be  instantiated  as  a  fighter  jet  with  a  weapons 
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capacity  of  10,000  pounds  and  a  maximum  speed  of  1,200  miles  per  hour  (a  =  fighter, 


10,000  pounds,  1,200  mph). 

DEFINITION  (System  Instantiation):  Given  a  specific  s  G  S  and  a  set  x  Q  X  of  system 
characters  descriptive  of  s,  then  a=x(s)  is  a  sequence  of  system  character  states, 
called  the  instantiation  of  s,  which  models  s.  (Ford,  2008) 

Ford‘s  interoperability  measurement  is  based  on  a  mathematical  similarity 
measurement.  To  put  it  simply,  a  similarity  measurement  is  based  upon  distance.  Items 
that  are  similar  have  less  distance,  and  items  that  are  disparate  have  more  distance 
between  them.  There  are  various  types  of  similarity  measurements,  and  Ford  chose  to 
use  the  Minkowski  similarity  function  to  base  his  analysis  of  interoperability 
measurement  upon.  This  function  enables  interoperability  measurement  between 
multiple  systems  based  upon  shared  interoperability  characters  that  have  been  instantiated 
and  aligned.  Ford  captures  the  SimRea\  interoperability  measurement  as  follows: 


Equation  1.  Average  Character  State  Value  (Ford,  2008) 


n 


n 


Average  Character  State  Value  =  w  =  — 


2  nc, 


i= 1 


max 


DEFINITION  ( SimReal):  Given  a  pair  of  systems  s',s"  instantiated  as 


a',  a"  e  "  n  [0,  cmax  ]  ,  then  I  -  SimReal  -  w-MMS ,  written  out 


completely  in  [equation  2]  is  an  interoperability  function  which  gives  a 


weighted,  normalized  measure  of  the  similarity  of  systems  instantiated 
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with  real-valued  character  states  where  w  is  the  average  character  state 


value  of  a  pair  of  system  instantiations,  MMS  is  the  Modified  Minkowski 
Similarity  function,  n  is  the  number  of  characters  used  to  instantiate 
a',  a" ,  cmax  is  the  maximum  character  state  value,  and  r  is  the 

Minkowski  parameter  (usually  set  to  r  —  2 ). 

Equation  2.  SimReai  Interoperability  Measurement  (Ford,  2008) 


1  =  SimRea,  = 


i= 1 


i=l 


2  nc„ 


1- 


\yfn  j 


2>, 

i=i 


f  < 


a[i)-a"[i) 


"'max  J 


Ford‘s  axiom  further  states: 

AXIOM  (System  Similarity  and  Interoperability) :  If  a  pair  of  systems  is 
instantiated  only  with  system  interoperability  characters,  then  the  measure  of 
their  similarity  is  also  a  measure  of  their  interoperability.  (Ford,  2008) 


Levels  of  Information  Systems  Interoperability  (LISI) 

The  LISI  model  was  designed  to  measure  information  systems1  interoperability  by 
detennining  the  degree  of  interoperability  achieved  by  systems  or  organizations  (Kasunic 
&  Anderson,  2004).  In  short,  it  is  a  maturity  model.  The  model  uses  four  main 
-attributes  of  interoperability”  (Kasunic  &  Anderson,  2004)  to  compose  the  framework 
used  to  measure  interoperability;  procedures,  applications,  infrastructure,  and  data,  also 
known  as  PAID. 
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Table  2.  LISI  Attributes  (Kasunic  &  Anderson,  2004) 


□ 

Procedures 

Policies  and  procedures  govern  a  system's  development  through  established  standards  and 
the  procedures  and  processes  which  influence  system  integration  and  functional 
operational  requirements 

□ 

Applications 

The  functions  a  system  is  intended  to  perform  These  functions  reside  most  often  in  the 
form  of  user-based  application  programs  which  perform  or  support  a  specific  set  of 
processes  or  procedures 

D 

Infrastructure 

The  infrastructure  required  to  support  the  systems  operations.  Contains  four  sub¬ 
components  which  are  also  defined  in  terms  of  increasing  levels  of  sophistication 

□ 

Data 

The  data  and  information  structures  used  to  support  both  the  functional  applications  and 
system  infrastructure 

The  PAID  attributes  make  up  the  columns  of  the  LISI  reference  model.  The  rows  in  the 
model  define  the  level  of  interoperability  maturity  and  are  defined  by  the  following  (in 
order  of  increasing  maturity);  isolated,  connected,  functional,  domain  and  enterprise. 
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Table  3.  LISI  Maturity  Levels  (Kasunic  &  Anderson,  2004) 


Infonnation  Exchange 

Level 

Computing  Environment 

Distributed  global  information  and 
applications 

Simultaneous  interactions  with  complex 
data 

Advanced  collaboration,  e  g.  interactive 
COP  update 

Event-triggered  global  database  update 

4 

Enterprise 

Interactive  manipulation: 
shared  data  and  applications 

Gtebal  Information  Spac«  _ 

1  -j\  «j|f 

Shared  databases 

Sophisticated  collaboration,  e  g.. 

Common  Operating  Picture 

3 

Domain 

Shared  data:  separate 
applications 

ttamcritor  *3 

wrwnontK  fi 

X 

Battle  Manager 

Heterogeneous  product  exchange 

Basic  collaboration 

Group  collaboration,  e  g.,  exchange  of 
annotated  imagery,  maps  with  overlays 

2 

Functional 

Minimal  common 
functions:  separate  data  and 
applications 

41  M 

JpSK — I 

Homogeneous  product  exchange,  e  g.. 

FM  voice,  tactical  data  links,  text  files, 
transfers,  message,  e-mail 

1 

Connected 

Bectromc  connection: 
separate  data  &  applications 

41  ^ 

Vjtr 

Manual  gateway,  e  g.,  diskette,  tape,  hard 
copy  exchange 

0 

Isolated 

Non-connected 

#  f 

The  combination  of  the  interoperability  attributes  with  the  interoperability  maturity  levels 
results  in  the  LISI  reference  framework: 
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Table  4.  LISI  Reference  Model  (Ford,  2008) 


LEVEL 

(Environment) 


Enterprise 

Level 

(Universal) 


Domain 
Level  3 

(Integrated) 


Functional 
Level  2 


(Distributed) 


Connected 

Level 

(Peer-to-Peer) 


Isolated 

Level 


(Manual) 


Interoperabilty  Attributes 

P 

A 

D 

Multi-National 

Enterprises 

Interactive 

(cross 

applications) 

Multi- 

Dmientional 

Topologies 

Cross- 

Enteipnse 

Models 

Cross  Government 

Enterprise 

DoD  Enterprise 

Full  Object 
Cut  &  Paste 

Euteipuse 

Model 

Domain 

Service  Agency 
Doctrine.  Procedures. 
Training,  etc. 

Shared  Data 

(e  g..  Situation  Displays 
Direct  DB  Exchanges) 

WAN 

DBMS 

Group  Collaboration 
(e.g_  While  Boards.  VTC) 

Domain 

Models 

Full  Text 

Cut  &  Paste 

Coininou 

Operating 

Environment 

(eg.,  DD-COE 
Level  5) 
Compliance 

Web  Browser 

LAN 

Program 

Models 

& 

Advanced 

Data 

Fonnats 

Basic  Operations 

Documents 

Bnefinss 

Pictures  &Maps 

Program 

Standard  Procedures. 
Training,  etc. 

Adv.  Messaging 

Messge  Parses 
E-mail  \v  Attachments 

NET 

Standards 

Complaint 

(eg.,  JTA) 

Basic  Messaging 

(e.g..  Unformatted  Tea 
E— mail  w  o  Attachments 

Two  Way 

Basic 

Data 

Formats 

Data  File  Transfer 

Security  Profile 

Simple  Interaction 

(eg.  Telemetiy. 
Remote  Access 

Text  Chatter  \bioe.  Fax) 

OneWay 

Media  Exchange 
Procedures 

N/A 

Removable 

Media 

Media 

Formats 

j  NATO 
,  j  Level  3 

Manual 

Re-entry 

Piivate 

Data 

Manual  i 

Access  !  NATO 
f  ontrok  •  Level  2 

j  NATO 
!  Level  1 

NO  KNOWN  INTEROPERABILITY 


Referencing  back  to  Ford‘s  method  and  the  standard  definition  of  interoperability, 
LISI  provides  a  way  to  qualify  how  well  systems  provide  and  use  services  between  one 
another  to  operate  effectively.  The  -attributes  of  interoperability,”  or  PAID,  describe  key 
characteristics  necessary  for  interoperability  to  occur.  In  other  words,  the  attributes  are 
the  means  by  which  the  services  are  provided;  they  enable  interoperability.  The  levels 
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provide  the  characters1  states  where  higher  levels  of  maturity  will  result  in  higher  levels 
of  interoperability. 

The  LISI  model  provides  a  good  way  to  measure  the  overall  interoperability 
maturity  of  a  particular  system.  It  does  not  however,  guarantee  interoperability  between 
systems.  Two  systems  possessing  identical  maturity  levels  as  defined  by  LISI  will  not 
interoperate  if  they  use  disparate  technology,  standards  or  interfaces.  As  Ford  points  out, 
the  LISI  model4 s  strength  lies  in  its  ability  to  facilitate  a  quantifiable  interoperability 
measurement  (Ford,  2008). 


Organizational  Interoperability  Maturity  Model  (OIM) 

The  model  known  as  OIM  was  born  from  and  uses  the  same  basic  structure  as  the 
LISI  model,  but  instead  deals  with  interoperability  between  organizations.  Instead  of  the 
PAID  attributes,  OIM  utilizes  the  attributes  of  preparedness,  understanding,  command 
style  and  ethos: 

Preparedness:  This  attribute  describes  the  preparedness  of  the  organisation  to 
interoperate.  It  is  made  up  of  doctrine,  experience  and  training. 

Understanding:  The  understanding  attribute  measures  the  amount  of 
communication  and  sharing  of  knowledge  and  infonnation  within  the  organisation 
and  how  the  infonnation  is  used. 

Command  Style:  This  is  the  attribute  that  describes  the  management  and 
command  style  of  the  organisation  -  how  decisions  are  made  and  how  roles  and 
responsibilities  are  allocated/delegated. 

Ethos:  The  ethos  attribute  is  concerned  with  the  culture  and  value  systems  of  the 
organisation  and  the  goals  and  aspiration  of  the  organisation.  The  level  of  trust 
within  the  organisation  is  also  included.  (Clark  &  Jones) 

The  levels  of  organizational  interoperability  maturity  again  follow  the  LISI  model's  lead: 


18 


Level  0  -  Independent  -  The  Level  0  interoperability  describes  the  interaction 
between  independent  organisations.  These  are  organisations  that  would  normally 
work  without  any  interaction  other  than  that  provided  by  personal  contact.  They 
are  likely  to  be  organisations  that  do  not  normally  share  common  goals  or  purpose 
but  that  may  be  required  to  interoperate  in  some  scenario  that  has  no  precedent. 
Essentially  the  arrangements  are  unplanned  and  unanticipated. 

Level  1  -  Ad  hoc  -  At  this  level  of  interoperability  only  very  limited 
organisational  frameworks  are  in  place  which  could  support  ad  hoc  arrangements. 
There  will  be  some  guidelines  to  describe  how  interoperability  will  occur  but 
essentially  the  specific  arrangements  are  still  unplanned.  There  will  be  some 
overarching  shared  goal  but  individual  organisation  aspirations  will  take 
precedence  and  the  organisations  remain  entirely  distinct. 

Level  2  -  Collaborative  -  The  collaborative  organisational  interoperability  level 
is  where  recognised  frameworks  are  in  place  to  support  interoperability  and 
shared  goals  are  recognized  and  roles  and  responsibilities  are  allocated  as  part  of 
on-going  responsibilities  however  the  organisations  are  still  distinct.  Training  is 
likely  to  have  taken  place  in  some  aspects  of  the  interworking  and  significant 
communication  and  sharing  of  knowledge  does  occur  but  the  home  organisations’ 
frameworks  still  have  a  significant  influence. 

Level  3  -  Integrated  -  The  integrated  level  of  organisational  interoperability  is 
one  where  there  are  shared  value  systems  and  shared  goals,  a  common 
understanding  and  a  preparedness  to  interoperate,  for  example,  detailed  doctrine  is 
in  place  and  there  is  significant  experience  in  using  it.  The  frameworks  are  in 
place  and  practised  however  there  are  still  residual  attachments  to  a  home 
organisation. 

Level  4  -  Unified  -  A  unified  organisation  is  one  in  which  the  organisational 
goals,  value  systems,  command  structure/style,  and  knowledge  bases  are  shared 
across  the  system.  The  organisation  is  interoperating  on  continuing  basis.  This  is 
really  the  ideal  level  where  there  is  no  impediment  in  the  organisational 
frameworks  to  full  and  complete  interoperation.  It  is  likely  to  occur  only  in  very 
homogeneous  organisations.  (Clark  &  Jones) 

The  organizational  attributes  and  maturity  levels  are  combined  to  fonn  the  OIM: 
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Table  5.  OIM  (Clark  &  Jones) 


Preparedness 

Understanding 

C  ommand  Style 

Ethos 

Unified 

C  omplete  -  normal  day-to- 
day  working 

Shared 

Homogeneous 

Uniform 

Combined 

Detailed  doctrine  and 
experience  m  using  it 

Shared  comms  and  shared 
knowledge 

One  chain  of  command  and 
interaction  with  home  org 

Shared  ethos  but  with 
influence  from  home  org 

Collaborative 

General  doctrine  in  place 
and  some  experience 

Shared  comms  and  shared 
knowledge  about  specific 
topics 

Separate  reporting  lines  of 
responsibility  overlaid  with 
a  single  command  chain 

Shared  purpose;  goals, 
value  system  significantly 
influenced  by  home  org 

Ad  hoc 

General  guidelines 

Electronic  comms  and 
shared  information 

Separate  reporting  lines  of 
responsibility 

Shared  purpose 

Independent 

No  preparedness 

Communication  via  phone 
etc 

No  interaction 

Limited  shared  purpose 

The  creators  of  OIM  note  that  LISI  is  — strogly  technological”  and  focuses  on 

system  and  technical  compatibility,”  and  that  OIM  was  created  to  — look  lathe  layers  of 

the  model  that  deal  with  organizational  issues”  (Clark  &  Jones).  They  further  delineate 

the  differences  between  interoperability  that  is  driven  by  process  versus  technology: 

Where  interoperability  has  been  driven  by  process,  the  focus  is  on  the  situation, 
the  people  and  commander's  intent. 

Where  interoperability  has  been  driven  by  technology,  the  focus  is  on  assets,  their 
properties  and  the  levels  of  compatibility  required. 

Therefore,  OIM  qualitatively  assesses  organizational  interoperability  by  measuring  the 

maturity  of  an  organization's  processes,  their  situation,  the  people  involved  and  the 

commander4  s  intent.  These  focus  areas  are  characterized  by  the  attributes  of 

preparedness,  understanding,  command  style  and  ethos.  Again,  the  attributes  are  the 

means  by  which  the  services  from  the  definition  of  interoperability  are  provided,  where 

increased  levels  of  maturity  in  each  attribute  area  fundamentally  result  in  an  increase  in 

interoperability.  While  the  attributes  of  preparedness,  understanding,  command  style  and 

ethos  are  highly  relevant  to  an  operational  commander  and  their  troops4  interoperation, 
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other  attributes  may  be  selected  to  properly  characterize  a  highly  collaborative,  non¬ 
combat  acquisition  system. 


Ford’s  Method  Applied  to  OIM 

Ford  applied  his  interoperability  measurement  to  the  maturity  model,  in  this  case 
OIM,  to  provide  a  way  to  quantifiably  measure  the  interoperability  between  coalition 
nations.  An  INTERFET  coalition  exercise  was  used  to  execute  the  measurement  where 
the  systems  ( S)  were  the  nations,  their  characters  ( X)  were  the  attributes  of  preparedness, 
understanding,  command  style,  and  ethos,  and  their  character  states  (C)  were  represented 
by  the  five  levels  of  the  maturity  model  (Ford,  2008): 

5  =  {AUS, US, NZ, Thai, Phil, ROK} 

X  =  { Preparedness ,  Understanding, Command  Style, Ethos} 

C  =  {0,12,3,4} 

Using  maturity  level  assessments  derived  from  the  INTERFET  exercise,  the  instantiation 
of  S,  X,  and  C  yielded: 

Table  6.  INTERFET  Instantiation  (Ford,  2008) 


Preparation 

Understanding 

Command  Style 

Ethos 

AUS 

2 

3 

3 

1 

US 

2 

3 

3 

1 

NZ 

2 

3 

3 

1 

Thai 

1 

1 

1 

1 

Phil 

1 

1 

1 

1 

ROK 

0 

1 

1 

1 
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Ford‘s  interoperability  method  is  then  easily  applied  to  determine  the  level  of 
interoperability  between  the  coalition  nations  during  the  INTERFET  exercise: 

Table  7.  INTERFET  Interoperability  Measurement  (Ford,  2008) 


A  US 

US 

NZ 

Thai 

Phil 

ROK 

A  US 

0.6 

0.6 

0.6 

0.3 

0.3 

0.2 

US 

0.6 

0.6 

0.6 

0.3 

0.3 

0.2 

NZ 

0.6 

0.6 

0.6 

0.3 

0.3 

0.2 

Thai 

0.3 

0.3 
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This  method  can  be  applied  to  any  interoperability  maturity  model  to  determine  how  well 
a  nation  or  system  interoperates  with  another. 

Space  Acquisition  Failures  and  Fessons  Fearned 

The  Department  of  Defense‘s  space  acquisition  community  has  suffered  through 
numerous  failures  as  documented  by  the  press,  and  perhaps  more  extensively,  the 
Government  Accountability  Office  (GAO).  It  is  important  to  briefly  define  what  is  meant 
by  a  failed  acquisition  program.  As  defined  by  Merriam- Webster,  failure  is  an  -emission 
of  occurrence  or  performance;  specifically,  a  failing  to  perform  a  duty  or  expected 
action.”  In  space  acquisition  terms,  a  failure  can  be  considered  a  failing  to  meet  cost, 
schedule  or  performance  objectives  as  originally  defined  by  the  program  at  its  inception. 

A  thorough  review  of  past  space  acquisition  failures  was  conducted.  The  majority 
of  inputs  were  authored  by  the  Government  Accountability  Office  (GAO),  who  has  an 
extensive  history  of  evaluating  space  acquisition  performance.  The  GAO  reports  were 
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frequently  corroborated  by  independent  reviews  and  other  sources,  and  revealed  shared 
reasons  for  space  acquisition  failures.  Perhaps  the  best  overall  summary  was  provided  by 
a  Defense  Science  Board  report  produced  in  2003.  The  group  was  chaired  by  Mr.  Tom 
Young,  a  fonner  Martin  Marietta  Chief  Executive,  and  included  membership  from  many 
space  acquisition  stalwarts. 

The  report,  simply  titled  -Acquisition  of  National  Security  Space  Programs”,  and 
despite  its  2003  publication  date,  -discerned  profound  insights  into  systemic  problems  in 
space  acquisition”  (Office  of  the  Undersecretary  of  Defense  for  Acquisition,  Technology 
and  Logistics,  2003)  that  still  resonate  within  the  space  acquisition  program  offices  of 
today.  The  report  highlighted  five  deep-seated  shortcomings  responsible  for  many  of  the 
space  acquisition  failures  of  the  past. 

First,  “cost  has  replaced  mission  success  as  the  primary  driver  in  managing 
space  development  programs.  ”  The  report  emphasizes  the  fragility  of  space  acquisition 
programs,  citing  how  -thousands  of  good  decisions  can  be  undone  by  a  single 
engineering  flaw  or  workmanship  error,  and  these  flaws  and  errors  can  result  in 
catastrophe”  (Office  of  the  Undersecretary  of  Defense  for  Acquisition,  Technology  and 
Logistics,  2003).  In  basic  terms,  trading  schedule  and  cost  savings  at  the  expense  of 
mission  success  is  a  flawed  approach.  For  space  acquisitions  programs,  approximately 
70%  of  the  life  cycle  cost  occurs  before  the  system  is  fielded: 
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Figure  3.  Space  Life  Cycle  Cost  Curve  (Hamel,  2007) 

There  is  great  pressure  to  maintain  cost  and  schedule  during  this  phase,  but  doing  so  at  an 
increased  level  of  mission  risk  ultimately  results  in  the  opposite  effect.  The  following 
figure  characterizes  a  typical  situation;  a  program  manager  chooses  to  save  budget  and 
schedule  by  using  mission  success  as  margin.  The  ultimate  effect  is  an  increased  risk  of 
mission  failure,  which  leads  to  greater  cost  increases  and  schedule  delays  in  the  future. 
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Figure  4.  Space  Acquisition  Scenario  (Office  of  the  Undersecretary  of  Defense  for 


Acquisition,  Technology  and  Logistics,  2003) 


Second,  “unrealistic  estimates  lead  to  unrealistic  budgets  and  unexecutable 
programs.  ”  This  root  cause  stems  primarily  from  optimistic  cost  estimates  provided  by 
an  incoming  competitor  and  inadequate  cost  margin.  The  report  states  that  -analysis  of 
recent  space  competitions  found  that  the  incumbent  contractor  loses  more  than  90  percent 
of  the  time.”  The  incumbent  contractor  is  -burdened”  by  its  very  relevant  and  real-world 
legacy  program  experience,  and  will  produce  more  realistic  cost  estimates  for  a  follow-on 
program.  The  incumbent  is  held  at  a  disadvantage  however,  to  a  new,  less  experienced 
contractor  who  may  produce  optimistic  cost  estimates  to  gain  an  edge  in  the  competition. 
If  the  less  experienced  contractor  wins  the  new  contract,  the  program  office  budget  is 
matched  to  the  unrealistic  estimate,  almost  guaranteeing  future  program  delays  and 
increased  risk  to  mission  success.  The  report  also  cites  an  unhealthy  emphasis  on 
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program  advocacy  at  the  expense  of  realism  as  a  major  contributor  to  unrealistic 
estimates. 

Third,  “undisciplined  definition  and  uncontrolled  growth  in  system  requirements 

increase  cost  and  schedule  delays.  ”  The  inherent  complexity  of  space  systems  and  the 

ever  increasing  numbers  of  users  have  driven  program  requirements  and  Key 

Perfonnance  Parameters  (KPP)  to  unmanageable  proportions.  The  report  adds,  -e-lear 

tradeoffs  among  cost,  schedule,  risk,  and  requirements  are  not  well  supported  by  rigorous 

system  engineering,  budget,  and  management  processes.”  Programs  with  increased 

numbers  of  users  and  KPPs  are  more  likely  to  suffer  cost  and  schedule  delays. 

Fourth,  “ government  capabilities  to  lead  and  manage  the  space  acquisition 

process  have  seriously  eroded.  ”  Inexperienced  program  managers  tend  to  adopt  a  -ean- 

do”  attitude  in  place  of  programmatic  rigor.  Additionally,  the  report  states: 

Policies  and  practices  inherent  in  acquisition  reform  inordinately  devalued  the 
systems  acquisition  engineering  workforce.  As  a  result,  today‘s  government 
systems  engineering  capabilities  are  not  adequate  to  support  the  assessment  of 
requirements,  conduct  trade  studies,  develop  architectures,  define  programs, 
oversee  contractor  engineering,  and  assess  risk.  With  growing  emphasis  on 
effects-based  capabilities  and  cross-system  integration,  systems  engineering 
becomes  even  more  important  and  interim  corrective  action  must  be  considered. 

A  less  experienced  acquisition  workforce  results  in  an  inability  to  manage  programs,  and 

very  clearly  results  in  program  failures. 

Fifth  and  finally,  “industry  has  failed  to  implemen  t  proven  practices  on  some 

programs.  ”  The  report  credits  industry4  s  knowledge  of  and  ability  to  utilize  proven 

practices,  but  gently  censures  them  for  a  lack  of  focus  and  dedication  to  the  same. 
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The  GAO‘s  2003  report,  -Military  Space  Operations:  Common  Problems  and 
their  Effects  on  Satellite  and  Related  Acquisitions,”  provides  a  similarly  useful  summary 
of  space  acquisition  failures  over  the  last  two  decades.  It  cites  four  primary  root  causes 
of  space  acquisition  failures,  and  corroborates  many  of  the  findings  from  the  2003 
Defense  Science  Board  report. 

First,  “ requirements  for  what  the  satellite  needed  to  do  and  how  well  it  must 
perform  were  not  adequately  defined  at  the  beginning  of  a  program  or  were  changed 
significantly  once  the  program  had  already  begun  ”  (Government  Accountability  Office, 
2003).  The  GAO  asserts  that  this  issue  caused  programs  difficulty  in  matching 
requirements  to  their  resources,  and  resulted  in  cost  and  schedule  increases.  This  finding 
matches  the  third  Defense  Science  Board  finding  listed  above. 

Second,  “investment practices  were  weak.  For  example,  potentially  more  cost- 
effective  approaches  were  not  examined  and  cost  estimates  were  optimistic  ” 
(Government  Accountability  Office,  2003).  The  GAO  specifically  cites  the  lack  of  an 
overall  investment  strategy  for  DOD  space  as  the  root  cause,  where  shifts  in  budget  were 
unexpected,  and  money  was  often  moved  from  healthier  programs  to  pay  for  weaker 
ones.  This  finding  corroborates  the  second  Defense  Science  Board  finding  above. 

Third,  “acquisition  strategies  were  poorly  executed.  For  example,  competition 
was  reduced  for  the  sake  of  schedule  or  DOD  did  not  adequately  oversee  contractors  ” 
(Government  Accountability  Office,  2003).  The  GAO  adds  that  the  DOD  took  a 
-schedule-driven  versus  a  knowledge-driven  approach  to  the  acquisition  process”  which 
strongly  supports  the  first  Defense  Science  Board  finding  listed  previously. 
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Fourth,  and  finally,  “technologies  were  not  mature  enough  to  be  included  in 

product  development”  (Government  Accountability  Office,  2003).  Although  not 

specifically  stated,  this  cause  is  tied  to  the  Defense  Science  Board‘s  fourth  finding  above. 

It  can  be  inferred  that  a  lack  of  program  management  and  systems  engineering  rigor 

allowed  immature  technologies  to  be  used,  ultimately  resulting  in  increased  program  risk. 

In  2006,  a  Defense  Acquisition  Perfonnance  Assessment  (DAP A)  panel  was 

commissioned  by  the  Deputy  Secretary  of  Defense  to  evaluate  the  overall  defense 

acquisition  enterprise.  The  panePs  major  findings  place  significant  emphasis  on  the 

ability  of  acquisition^  major  elements  to  interoperate  effectively: 

The  evidence  we  discovered  was  persistent  in  recognizing  that  an  effective 
Acquisition  System  requires  stability  and  continuity  that  only  can  be  provided 
through  successful  integration  of  the  major  elements  upon  which  it  depends. 
When  we  began  this  task,  we  presumed  the  Department1  s  Acquisition  System  to 
be  an  efficient  integration  of  the  acquisition,  requirements  and  budget  processes. 
However,  in  the  course  of  our  review  we  found  that  the  System  is  a  highly 
complex  mechanism  that  is  fragmented  in  its  operation.  (Defense  Acquisition 
Perfonnance  Assessment  Project,  2006) 

The  report  breaks  defense  acquisition  into  six  major  elements;  organization,  workforce, 
budget,  requirements,  acquisition  [process],  and  industry,  and  provides  an  assessment  of 
each  element.  Although  not  specific  to  space  acquisition,  the  DAPA  review  generally 
upholds  the  findings  from  the  2003  Defense  Science  Board  report. 

The  panel  found  that  “the  current  decision  making  process  is  flawed”  in  the 
organization  element  introducing  uncertainty  and  instability  in  program  execution.  The 
panel  cites  several  root  causes  including  disconnects  between  the  acquisition,  budget  and 
requirements  processes.  This  finding  is  captured  at  a  high-level,  and  can  be  attributed  to 
many  of  the  findings  from  the  2003  Defense  Science  Board‘s  report. 
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For  the  workforce  element,  the  panel  noted  that  “a  successful  program  requires  a 

professional  workforce  with  subject  matter  expertise.  ”  The  panel  alludes  to  poor 

integration  of  budget  and  requirements  personnel  into  the  acquisition  workforce,  as  well 

as  a  significant  lack  of  experience  and  training  as  primary  workforce  problem  areas.  This 

corresponds  to  the  fourth  Defense  Science  Board  finding  above.  The  panel  also  states: 

Experience  and  expertise  in  all  functional  areas  has  been  de-valued  and 
contributes  to  a  -Conspiracy  of  Hope”  in  which  we  understate  cost,  risk  and 
technical  readiness  and,  as  a  result,  embark  on  programs  that  are  not  executable 
within  initial  estimates.  (Defense  Acquisition  Performance  Assessment  Project, 
2006) 

The  -Conspiracy  of  Hope”  strongly  supports  the  first  and  second  Defense  Science  Board 
findings  listed  previously. 

For  the  budget  element,  the  panel  emphasizes  that,  “successful  Research, 
Development,  Test  and  Evaluation  and  Procurement  programs  require  stable  budgets 
and  accurate  planning’’  and  that  this  stability  simply  does  not  exist.  The  panel  also 
detennined  that  budget  shortfalls  were  often  met  by  -stretching  programs”  whereby  the 
DOD  -accepts  long-tenn  cost  increases  and  delays  in  acquisition  programs  to  achieve 
short-term  savings  and  budget  flexibility.”  These  findings  directly  support  the  Defense 
Science  Board‘s  first  finding  above.  The  panel  also  cites  a  problematic  use  of  optimistic 
budget  estimates  on  acquisition  programs,  further  supporting  the  Defense  Science 
Board‘s  second  finding. 

The  panel  continues  by  listing  a  “lengthy  and  insufficiently  advised  requirement 
development  process  ”  often  based  upon  “immature  technologies  and  overly  optimistic 
estimates  of  future  resource  needs  and  availability  ”  as  well  as  requirements  instability  as 
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major  problem  areas  in  the  requirements  element.  These  findings  validate  the  second  and 
third  findings  from  the  Defense  Science  Board  report. 

The  acquisition  element,  when  viewed  from  a  process  perspective,  is 
characterized  by  the  panel  as  being  part  of  the  -Conspiracy  of  Hope”  where  “industry  is 
encouraged  to  propose  unrealistic  cost,  optimistic  performance  and  understate  technical 
risk  estimates  during  the  acquisition  solicitation  process  and  the  Department  is 
encouraged  to  accept  these  proposals  as  the  foundation  for  program  baselines,  ” 
reinforcing  the  Defense  Science  Board‘s  second  finding. 

The  panel  found  that  the  final  element,  industry,  is  extremely  fragile,  where 
“consolidation  of  the  industrial  base,  caused  by  unstable  defense  demand,  has  reduced 
the  benefits  of  competition,  introduced  industrial  organizational  conflict  of  interest 
issues,  and  made  every  defense  contract  a  “must  win  ”  situation  for  the  prime 
contractors.  ”  This  description  does  not  directly  relate  to  a  specific  Defense  Science 
Board  finding,  but  can  be  cited  as  a  contributing  factor. 

To  summarize,  the  DAP  A  findings  provide  a  more  recent  overview  of  acquisition 
lessons  learned  that  validate  the  earlier  Defense  Science  Board  findings  from  2003. 

Leading  Indicators 

The  -Systems  Engineering  Leading  Indicators  Guide”  serves  as  the  primary 
reference  for  leading  indicators  in  this  thesis.  It  states  that  — heading  indicator  is  a 
measure  for  evaluating  the  effectiveness  of  how  a  specific  activity  is  applied  on  a  project 
in  a  manner  that  provides  infonnation  about  impacts  that  are  likely  to  affect  the  system 
perfonnance  objectives”  and  is  useful  to  -support  the  effective  management  of  systems 
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engineering  by  providing  visibility  into  expected  project  performance  and  potential  future 
states”  (Massachusetts  Institute  of  Technology,  INCOSE,  and  PSM,  2010)1.  Leading 
indicators  often  use  historical  data  and  trends  to  provide  this  predictive  measure  of 
performance  and  are  -eomposed  of  characteristics,  a  condition  and  a  predicted  behavior”: 


Strength  Scale 


Figure  5.  Depiction  of  a  Leading  Indicator  (Massachusetts  Institute  of  Technology, 

INCOSE,  and  PSM,  2010) 


The  characteristics  are  the  data,  the  condition  supplies  the  context  in  which  to  evaluate 
the  data,  and  when  combined,  they  provide  an  indication  or  predicted  behavior.  The 
strength  scale  captures  the  relative  importance  of  the  indicator  based  upon  past 
experience. 

The  guide  recommends  that  leading  indicators  supplement  a  program1  s  existing 
set  of  measures,  and  urges  them  to  properly  set  planned  targets  and  thresholds  using 


1  General  Use:  Permission  to  reproduce,  use  this  document  or  parts  thereof,  and  to  prepare  derivative 
works  from  this  document  is  granted,  with  attribution  to  LAI,  INCOSE,  PSM,  and  SEAri,  and  the  original 
author(s),  provided  this  copyright  notice  is  included  with  all  reproductions  and  derivative  works. 
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empirical  data.  If  data  is  not  available,  expert  opinion  may  be  used  to  set  targets.  The 
preferred  method  however  is  to  utilize  -a-  good  historical  base  of  information”  where 
-organizations. .  .build  the  collection  of  the  historical  measurement  data  into. .  .collection 
practices”  (Massachusetts  Institute  of  Technology,  INCOSE,  and  PSM,  2010). 

These  targets  are  based  upon  the  phases  of  the  acquisition  cycle  as  defined  by 
DOD  Instruction  5000.02,  and  help  programs  understand  readiness  with  respect  to 
program  milestones.  The  applicability  of  the  indicator  also  varies  with  program  phase. 
As  illustrated  by  the  following  figure,  the  indicator  -Requirements  Trends”  provides 
insight  in  all  acquisition  phases,  while  -System  Definition  Change  Backlog  Trend”  is 
only  applicable  to  three  of  the  five  phases. 


Table  8.  Systems  Engineering  Leading  Indicators  Overview  (truncated) 
(Massachusetts  Institute  of  Technology,  INCOSE,  and  PSM,  2010) 
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Requirements 

Trends 

Rate  of  maturity  of  the  system  definition  against  the 
plan.  Additionally,  characterizes  the  stability  and 
completeness  of  the  system  requirements  that  could 
potentially  impact  design,  production,  operational  utility, 
or  support. 

System 

Definition 
Change  Backlog 
Trend 

Change  request  backlog  which,  when  excessive,  could 
have  adverse  impact  on  the  technical,  cost  and  schedule 
baselines. 

• 

• 

• 

• 

• 

• 

Interface 

Trends 

Interface  specification  closure  against  plan.  Lack  of 
timely  closure  could  pose  adverse  impact  to  system 
architecture,  design,  implementation  and/or  V&V  any  of 
which  could  pose  technical,  cost  and  schedule  impact. 

Requirements 

Validation 

Trends 

Progress  against  plan  in  assuring  that  the  customer 
requirements  are  valid  and  properly  understood. 

Adverse  trends  would  pose  impacts  to  system  design 
activity  with  corresponding  impacts  to  technical,  cost  & 
schedule  baselines  and  customer  satisfaction. 
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The  guide  provides  18  major  leading  indicators  with  example  graphs  and  detailed 
specifications  for  use  on  acquisition  programs.  The  -Requirement  Trends”  display  below 
demonstrates  how  a  leading  indicator  might  be  used  on  an  acquisition  program: 


Requirements  Trends 


Requirements  Growth  Trends 


Figure  6.  Requirements  Trends  Example  (Massachusetts  Institute  of  Technology, 

INCOSE,  and  PSM,  2010) 

In  this  example,  the  leading  indicator  shows  a  problem  with  requirements  growth 
which  drives  the  program  to  take  action  in  April.  Later  that  summer,  the  program  was 
able  to  evaluate  the  effectiveness  of  their  actions  as  the  growth  in  requirements  was 
stemmed.  The  -Requirements  Trends”  leading  indicator  can  be  directly  attributed  to  the 
2003  Defense  Science  Board‘s  finding  -undisciplined  definition  and  uncontrolled  growth 
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in  system  requiremen  ts  increase  cost  and  schedule  delays  ”  (Office  of  the  Undersecretary 
of  Defense  for  Acquisition,  Technology  and  Logistics,  2003),  and  provides  a  way  to 
measure  how  well  a  program  is  or  is  not  protecting  itself  from  the  causes  of  the 
acquisition  failures  of  the  past. 

It  is  important  to  recognize  that  lessons  learned  represent  a  source  of  historical 
empirical  data  that  can  be  used  to  develop  a  leading  indicator.  In  this  case,  the  Defense 
Science  Board,  GAO  and  DAPA  reports  provide  the  characteristics  and  conditions  in 
which  to  formulate  and  evaluate  a  leading  indicator  of  space  acquisition  program  success. 

Summary 

The  2003  Defense  Science  Board  and  GAO  reports  provide  a  concise  and 
comprehensive  review  of  the  space  acquisition  failures  of  the  past.  Each  report  supports 
the  other‘s  findings,  and  presents  a  strong  case  identifying  the  primary  space  acquisition 
characteristics  that  drive  success  or  failure.  These  characteristics  can  serve  as  attributes 
or  characters  for  the  basis  of  an  interoperability  measurement  and  a  leading  indicator  of 
program  perfonnance. 
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III.  Methodology 


Furthermore,  if  systems  implement  a  confrontational  operational  process  and  are  identified 
and  modeled  in  the  context  of  a  measure  of  operational  effectiveness  tied  to  that  process,  then 
another  fundamental  result  mathematically  relates  the  change  in  interoperability  of  the 
systems  with  a  change  in  the  measure  of  operational  effectiveness. 

— Thomas  C.  Ford  (Ford,  2008) 

Chapter  Overview 

The  purpose  of  this  chapter  is  to  develop  a  method  to  measure  acquisition 
interoperability  using  key,  driving  acquisition  characters  derived  from  past  acquisition 
experience.  This  method  will  use  Ford‘s  inaugural  work  on  confrontational 
interoperability  measurement  to  create  a  new  collaborative,  acquisition  interoperability 
measurement,  and  will  be  applied  to  several  space  acquisition  programs  for 
demonstration  in  the  -Analysis  and  Results”  chapter.  These  scenarios  will  also 
emphasize  the  utility  of  the  new  method  for  assessing  the  effectiveness  of  acquisition 
processes  that  must  interact  to  accomplish  a  specific  goal. 

Space  Acquisition  Interoperability 

Within  an  acquisition  program,  there  is  little  doubt  that  cost,  schedule  and 
requirements  are  intrinsically  linked.  The  numerical  values  of  cost  in  dollars,  schedule  in 
years,  and  requirements  in  number  of  KPPs  can  be  approximated  using  basic 
mathematical  relationships,  e.g.  more  KPPs  will  result  in  an  increase  in  cost  and 
schedule.  Acquisition  systems  on  the  other  hand  drive  the  ability  of  a  program  to  manage 
and  control  these  values.  The  people,  processes  and  products  that  detennine  a  program1  s 
cost,  schedule  and  requirements  can  be  described  as  systems.  The  following  figure  from 
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the  Defense  Science  Board  report  supplies  a  depiction  of  the  major  space  acquisition 
systems  found  on  a  typical  space  program  (Office  of  the  Undersecretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  2003): 


Requirements 


Schedule 


Margin 


Cost 


Launch 


Figure  7.  Space  Acquisition  Systems  (Office  of  the  Undersecretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  2003) 


Note:  In  this  case,  the  term  “margin  ”  is  not  a  system  in  and  of  itself.  It  is  better  described  as  a 
program  characteristic.  Additionally,  the  report  included  the  term  “launch  ”  to  illustrate  specific 
launch  constraints  placed  upon  a  program,  e.g.  the  satellite ’s  size  and  weight.  Although  launch  is 
an  important  aspect  of  any  program,  it  was  not  explicitly  cited  as  a  prime  cause  of  space 
acquisition  failures  and  was  omitted  as  a  result. 


These  acquisition  systems  must  work  together  to  create  a  program  that  is  executable  in 
order  to  deliver  mission  performance  on  time  and  budget.  More  specifically,  the  cost, 
schedule  and  requirements  systems  must  provide  and  accept  services  from  one  another  in 
order  to  operate  effectively.  Therefore,  using  Ford‘s  nomenclature,  the  space  acquisition 
systems  are  captured  as:  S  =  {cost,  schedule,  requirements}.  Each  system  represents  the 
people,  processes  and  products  that  determine  and  control  the  cost,  schedule  and 
requirements  for  a  given  program. 


36 


The  interoperations  that  occur  between  these  space  acquisitions  systems  are 
profuse  and  complex.  Using  the  lessons  learned  from  past  space  acquisition  failures,  the 
quintessential  interoperations  can  be  extracted  and  applied  to  a  maturity  model.  In  order 
to  compose  the  model,  the  individual  interoperations  between  systems  must  be  examined. 

Requirements-to-Schedule  Interoperability 

The  requirements-to-schedule  interoperation  is  most  simply  described  in  terms  of 
requirements  stability,  scope  and  complexity.  As  the  2003  Defense  Science  Board  report 
states,  more  Key  Performance  Parameters  (KPP)  as  well  as  — udisciplined  definition  and 
uncontrolled  growth”  in  requirements  will  result  in  increased  schedule  (Office  of  the 
Undersecretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  2003).  More 
complex  requirements  may  drive  a  program  to  pursue  less  mature  technologies,  which 
will  also  result  in  increased  schedule.  The  schedule  system  cannot  effectively  operate  if 
the  requirements  system  is  unstable  or  attempts  to  take  major  leaps  in  capability. 

ScheduIe-to-Requirements  Interoperability 

This  interoperation  is  focused  on  the  program1  s  intent  and  priority  when 
perfonning  trades  or  utilizing  margin.  A  program1  s  requirements  become  less  executable 
when  schedule  is  traded  at  the  expense  of  mission  assurance.  Therefore,  this  relationship 
is  defined  by  the  way  the  systems  utilize  shared  margin,  and  how  that  use  impacts  the 
progranTs  overall  executability.  If  the  schedule  system  dominates  trade  decisions,  the 
requirements  system  will  be  compromised. 
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Requirements-to-Cost  Interoperability 

The  requirements-to-cost  interoperation  is  analogous  to  the  requirements-to- 
schedule  interoperation.  More  KPPs  drive  program  complexity  and  decrease  the  ability 
of  the  cost  system  to  accurately  predict  and  control  program  costs.  Again,  the 
requirements  drive  the  selection  of  technology.  That  is,  a  demand  for  cutting  edge 
perfonnance  will  likely  drive  a  state-of-the-art  technology  choice,  which  decreases  cost 
confidence.  Additionally,  the  requirements  influence  the  level  of  experience  that  exists 
within  industry.  If  the  requirements  system  chooses  to  baseline  a  program  that  has  no 
legacy  or  comparable  system,  then  the  availability  of  experience  will  shrink,  ultimately 
decreasing  cost  confidence. 

Cost-to-Requirements  Interoperability 

Unrealistic  cost  estimates  have  been  a  mainstay  of  space  acquisition  criticism  and 
are  the  driving  factor  in  this  interoperation.  More  realistic  cost  estimates  will  deliver  a 
program  that  is  more  executable.  The  ability  of  the  cost  system  to  justify  and  secure  the 
appropriate  funding  will  also  influence  the  requirements  system1  s  ability  to  achieve  the 
intended  perfonnance. 

Schedule-to-Cost  Interoperability 

The  schedule-to-cost  interoperation  again  deals  with  realistic  estimates  and  how 
margin  is  utilized  by  the  acquisition  systems.  In  this  case,  a  poor  schedule  estimate  or 
overly  aggressive  plan  will  result  in  a  poor  cost  estimate.  When  schedule  is  traded  at  the 


38 


expense  of  mission  assurance,  the  cost  system  will  suffer  because  it  is  forced  to  accept 
more  risk. 

Cost-to-Schedule  Interoperability 

This  interoperation  is  simply  the  reverse  of  the  schedule-to-cost  interoperation. 
Therefore  the  schedule-to-cost  or  cost-to-schedule  relationship  is  considered  to  be  bi¬ 
directional. 

Space  Acquisition  Interoperability  Characters 

The  National  Security  Space  Acquisition  Policy  Interim  Guidance  provides  a 
good  starting  point  for  character  development.  Paragraph  4,  titled  -BOD  SPACE  MDA 
GUIDING  PRINCIPLES”  states,  — ©er  the  first  fifty  years  of  the  history  of  space 
acquisition,  several  enduring  principles  have  emerged.  The  following  principles  should 
be  considered  by  all  NS  S  members  to  set  the  tone  and  guide  decision  making  in  the 
acquisition  of  NSS  systems”  (Department  of  Defense,  2009).  When  merged  with  the  key 
system  interoperations  based  upon  past  acquisition  failures,  the  vital  interoperability 
characteristics  that  must  be  present  and  mature  to  enable  mission  success  are  revealed. 

As  captured  in  the  interim  guidance,  the  principles  of  -mission  success”,  -stable”, 
-disciplined”  and  -eost  realism”  best  match  the  findings  from  the  2003  Defense  Science 
Board  report: 

Mission  Success:  The  overarching  principle  behind  all  National  Security  Space 
programs  is  mission  success.  When  acquiring  space  systems,  mission  success 
must  be  the  first  consideration  when  assessing  the  risks  and  trades  among  cost, 
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schedule,  and  performance.  Risk  management,  test  planning,  system  engineering, 
and  funding  profdes  must  be  driven  by  this  objective. 

Stable:  Within  a  given  acquisition  increment,  stable  budgets,  stable 
requirements,  stable  direction,  and  low  personnel  turnover  are  necessary  for 
successful  program  acquisition.  Decisions  made  by  the  acquisition  execution 
chain  must  be  durable. 

Disciplined:  All  parties  to  this  space  acquisition  policy  must  exercise  the 
discipline  necessary  to  achieve  its  goals  without  allowing  its  procedures  to 
become  unnecessarily  burdensome  and/or  time  consuming. 

Cost  Realism:  The  goal  is  to  develop  and  grow  a  world-class  national  security 
space  cost  estimating  capability.  Cost  estimates  must  be  independent  and 
accomplished  in  a  timely,  realistic,  and  complete  manner.  Cost  will  be  controlled 
by  estimating  accurately  and  focusing  on  quality  to  reduce  rework  and  achieve 
mission  success.  All  members  of  the  NSS  acquisition  execution  chain  must  insist 
on,  and  protect,  a  realistic  management  reserve. 

These  NSS  principles  are  then  tailored  for  the  purpose  of  measuring  acquisition 
interoperability  based  upon  the  cardinal  lessons  learned.  The  following  four  acquisition 
interoperability  characters  and  definitions  are  produced: 

Mission  Focus:  When  acquiring  space  systems,  mission  success  must  be  the  first 
consideration  when  assessing  the  risks  and  trades  among  cost,  schedule,  and 
perfonnance.  Risk  management,  test  planning,  system  engineering,  and  funding 
profiles  must  be  driven  by  this  objective. 
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Stability:  Within  a  given  acquisition  increment,  stable  budgets,  stable 
requirements,  stable  direction,  and  low  personnel  turnover  are  necessary  for 
successful  program  acquisition.  Decisions  made  by  the  acquisition  execution 
chain  must  be  durable.  Technology  is  sufficiently  mature  and  stable. 

Discipline:  The  program  must  exercise  the  discipline  necessary  to  achieve  its 
goals  using  an  experienced  acquisition  team  adhering  to  proven  programmatic  and 
system  engineering  processes. 

Realism:  Program  estimates  must  be  independently  verified  and  accomplished  in 
a  timely,  realistic,  and  complete  manner.  Technology,  schedule  and  cost  will  be 
controlled  by  estimating  accurately  and  focusing  on  quality  to  reduce  rework  and 
achieve  mission  success.  All  members  of  the  NSS  acquisition  execution  chain 
must  insist  on,  and  protect,  a  realistic  management  reserve. 

These  four  characters,  X  =  {mission  focus,  stability,  discipline,  realism},  capture  the  chief 
causes  of  space  acquisition  failures  as  described  above  at  a  high  level.  It  is  recognized 
that  lower  levels  of  detail  will  discover  additional,  more  specific  characters  that  could 
lend  themselves  to  a  more  detailed  measure  of  acquisition  interoperability.  As  Ford 
noted  with  OIM,  -Although  the  final  version  of  the  OIM  model  remained  limited  to  a  4- 
attribute,  5 -level  model,  at  least  35  sub-attributes  were  further  defined”  however,  -by  not 
addressing  them  as  individual  attributes,  fidelity  is  lost  from  the  model,  and  their  contribution 
is  effectively  averaged  out”  (Ford,  2008).  Therefore  it  is  important  to  only  select  the  key  and 
driving  characters  as  evidenced  by  past  experience  to  avoid  watering  down  the  results  with 
less  relevant  characters. 
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Incorporating  Interoperability  Lessons  Learned  Into  a  Maturity  Model 

The  2003  GAO  and  Defense  Science  Board  reports  on  the  space  acquisition 
enterprise  provide  the  foundation  for  development  of  a  space  acquisitions  maturity  model 
based  upon  the  systems  and  characters  described  above.  The  levels  of  the  maturity  model 
can  be  built  by  examining  the  recommended  solutions  from  the  2003  Defense  Science 
Board  report  (Office  of  the  Undersecretary  of  Defense  for  Acquisition,  Technology  and 
Logistics,  2003)  discussed  earlier. 

Maturity  Model  Levels 

As  illustrated  earlier,  the  LISI  and  OIM  models  utilized  a  five-level  scale  to 
define  and  measure  the  maturity  of  a  system  in  a  specific  character.  The  acquisition 
model  will  follow  a  similar  five-level  construct  and  is  guided  by  the  definitions  provided 
by  OIM.  Using  Ford‘s  nomenclature,  the  maturity  levels  represent  character  states;  that 
is,  C  =  {0,  1,  2,  3,  4}. 

Level  0  -  Separated:  The  system  operates  independently  of  others.  The 
system1  s  goals  are  not  congruent  with  others,  and  little  evidence  exists  to  guide  decision¬ 
making.  The  people,  processes  and  products  of  the  system  are  inexperienced  and 
unproven. 

Level  1  -  Aligned:  The  system  recognizes  the  impact  it  has  on  others  and 
considers  space  acquisition  best  practices  and  lessons  learned  when  making  decisions,  but 
continues  to  value  its  own  goals  over  others.  The  people,  processes  and  products  possess 
experience  in  the  domain,  but  remain  unproven  for  the  task  at  hand. 
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Level  2  -  Structured:  The  system  is  aware  of  the  impact  it  has  on  others  and 
utilizes  space  acquisition  best  practices  and  lessons  learned  when  making  decisions.  The 
systenTs  decisions  are  made  with  a  shared  goal  in  mind.  The  people,  processes  and 
products  have  experience  executing  a  similar  task. 

Level  3  -  Associated:  The  system  understands  its  impact  on  others  and  has 
incorporated  space  acquisitions  best  practices  and  lessons  learned  into  its  overall 
baseline,  as  well  as  its  decision  making  process.  Decisions  are  made  in  a  structured 
environment  with  other  systems  using  shared  goals.  The  system  will  forgo  its  own 
interests  for  the  betterment  of  the  whole.  The  people,  processes  and  products  have 
executed  similar  tasks  on  many  occasions. 

Level  4  -  Accordant:  The  systenTs  decision-making  process  is  integrated  with 
others.  Space  acquisition  best  practices  and  lessons  learned  are  unified  across  all  systems 
and  guide  decision  making.  Knowledge  is  shared  and  understood  across  systems.  The 
system  does  not  maintain  its  own  interests;  all  efforts  are  integrated  toward  a  shared  goal. 
The  people,  processes  and  products  have  successfully  executed  nearly  identical  tasks  on 
many  occasions. 

The  Space  Acquisition  Interoperability  Maturity  Model  (SAIMM) 

The  SAIMM  fuses  the  space  acquisition  interoperability  characters  and  maturity 
levels  into  a  format  that  readily  facilitates  an  acquisition  interoperability  measurement 
based  upon  space  acquisition  history.  The  character  states  capture  the  key  and  driving 
aspects  of  each  character,  and  reflect  the  significant  drivers  of  the  cost,  schedule  and 
requirements  systems1  interoperation  success. 
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Table  9.  SAIMM 


MISSION  FOCUS 

STABILITY 

DISCIPLINE 

REALISM 

Level  4- 

Accordant 

Mission  success  is  weighted 
significantly  higherthan  cost  and 
schedule;  its  priority  is  reflected  in  risk 
management,  test  planning,  system 
engineering,  and  funding  profiles 

Stable  budgets,  stable  requirements, 
stable  direction,  and  low  personnel 
turnover  are  maintained  across  the 
board;  program  decisions  are  constant; 
technology  is  mature  (TRL  7  or  higher) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  prime  contractor)  and 
maintains  strict  adherence  to  proven 
programmatic  and  system  engineering 

processes 

Program  estimates  are  independently 
verified  and  accomplished  in  a  timely, 
realistic,  and  complete  manner; 
minimal  rework  is  required;  program 
funding  and  management  reserve  is 
realistic  (80/20  cost  confidence  with 
25%  management  reserve) 

Level  3- 

Assoclated 

Mission  success  is  weighted  higher 
than  cost  and  schedule 

Budgets,  requirements,  direction  and 
personnel  turnover  rarely  change  and 
remain  stable  in  critical  areas;  program 
decisions  rarely  change;  technology  is 
sufficiently  mature  (TRL  6) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  major  subcontractor)  and 
maintains  adherence  to  proven 
programmatic  and  system  engineering 

processes 

Program  estimates  are  independently 
verified;  rework  is  rarely  required; 
program  funding  is  unrealistic 
(between  80/20  and  50/50  cost 
confidence)  with  realistic  management 
reserve  (20%  or  greater) 

Level  2- 

Structured 

Mission  success  is  weighted  equally  to 

cost  and  schedule 

Budgets,  requirements,  direction  or 
personnel  turnover  change  occasionally 
in  a  few  minor  areas;  program  decisions 
change  occasionally;  technology 
requires  maturation  (TRL  5) 

The  acquisition  team  has  some 
experience  (previous  experience  on  a 
similar  program  as  a  subcontractor)  and 
utilizes  proven  processes  in  most  areas 

Program  estimates  are  independently 

verified  but  unrealistic  in  a  few  minor 
areas;  rework  is  occasionally  required; 
program  funding  is  unrealistic  (50/50 
cost  confidence)  with  realistic 
management  reserve  (20%  or  greater) 

Level  1- 
Allgned 

Cost  and  schedule  are  weighted  higher 
than  mission  success 

Budgets,  requirements,  direction  or 
personnel  turnover  change  frequently 
in  several  critical  areas;  program 
decisions  change  frequently; 
technology  requires  significant 
maturation  (TRL 4) 

The  acquisition  team  has  limited 
experience  (previous  experience  on  a 
portion  of  a  similar  program)  and  has 
knowledge  of  proven  processes  but 
does  not  use  them  consistently 

Program  estimates  are  unrealistic  in 
several  critical  areas;  rework  is 
frequently  required;  program  funding 
is  unrealistic  with  inadequate  (less 
than  20%)  management  reserve 

Level  0- 
Separated 

Cost  and  schedule  are  weighted 
significantly  higherthan  mission 

success 

Budgets,  requirements,  direction  and 
personnel  turnover  continually  change 
in  many  critical  areas;  program 
decisions  continually  change; 
technology  is  not  mature  (TRL  3) 

The  acquisition  team  has  no  experience 
(first-ever  program)  and  does  not  have 
access  to  proven  processes  (they  do  not 
exist) 

Program  estimates  are  unrealisticand 
focused  on  program  advocacy;  rework 
is  continually  required;  program 
funding  is  unrealistic  with  little  or  no 
management  reserve 

Note:  The  stability  character  also  captures  the  legal  requirement  contained  in  Title  10  United  States  Code  (U.S.C.)  Section  2366b  that  “ the 
technology’  in  the  program  has  been  demonstrated  in  a  relevant  environment  [commonly  accepted  as  technology  readiness  level  (TRL)  6],  as 
determined  by  the  Milestone  Decision  Authority  on  the  basis  of  an  independent  review  and  assessment  by  the  Director  of  Defense  Research  and 
Engineering  ” 
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Summary 

The  SAIMM  may  be  used  to  characterize  the  interoperability  maturity  of  any 
space  acquisition  program  by  providing  a  framework  infonned  by  the  acquisition  failures 
of  the  past. 


45 


IV.  Analysis  and  Results 


Recent  operations  have  once  again  illustrated  the  degree  to  which  U.S.  national  security 
depends  on  space  capabilities.  We  believe  this  dependence  will  continue  to  grow,  and  as 
it  does,  the  systemic  problems  we  identify  in  our  report  will  become  only  more  pressing 

and  severe. 

—  Mr.  A.  Thomas  Young  (Office  of  the  Undersecretary  of  Defense  for  Acquisition, 

Technology  and  Logistics,  2003) 

Chapter  Overview 

Using  the  OIM  construct  and  INTERFET  exercise,  Ford‘s  interoperability 
measurement  provided  insight  into  coalition  nation  interoperability.  The  SAIMM 
construct  delivers  a  mechanism  to  do  the  equivalent  for  the  space  acquisition  world. 

Space  acquisition  systems,  S  =  {cost,  schedule,  requirements},  can  be  characterized  using 
the  characters,  X  =  {mission  focus,  stability,  discipline,  realism},  and  character  states,  X 
=  {0,  1,  2,  3,  4},  from  SAIMM,  which  facilitates  an  interoperability  measurement  using 
Ford‘s  method.  In  this  chapter,  several  space  acquisition  programs  will  be  evaluated  for 
their  acquisition  systems1  interoperability  using  SAIMM.  A  review  of  each  evaluation 
will  be  used  to  explain  the  utility  of  the  acquisition  interoperability  measurement. 

It  is  important  to  note  that  some  subjectivity  was  required  when  evaluating  the 
programs  in  the  following  examples  using  SAIMM.  Each  SAIMM  level  includes 
multiple  factors  for  evaluating  a  program4  s  maturity.  For  example,  the  SAIMM  Level  3 
in  the  stability  character  requires  that  -budgets,  requirements,  direction  and  personnel 
turnover  rarely  change  and  remain  stable  in  critical  areas;  program  decisions  rarely 
change;  technology  is  sufficiently  mature.”  A  program  could  have  mature  technology  but 
suffer  from  requirements  that  frequently  changed.  The  resulting  SAIMM  score  in  this 
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character  could  fall  somewhere  between  1  and  3.  In  this  and  the  following  examples,  the 
worst  case  score  was  generally  chosen.  Scoring  was  further  influenced  by  the  historical 
result  and  impact  that  the  action  or  condition  had  on  the  program. 

Advanced  Extremely  High  Frequency  (AEHF)  System 

The  AEHF  system,  the  replacement  for  the  aging  MILSTAR  constellation  of 
communications  satellites,  recently  launched  the  first  of  four  satellites  into  orbit.  The 
prime  contractor  for  the  system  is  Lockheed  Martin  Space  Systems  Company  (Sunnyvale 
California)  and  the  payload  provider  is  Northrup  Grumman  Aerospace  Systems  (Redondo 
Beach  California).  Lockheed  Martin  Space  Systems  Company  was  also  the  prime 
contractor  on  the  legacy  MILSTAR  system. 

AEHF  was  challenged  by  multiple  launch  slips  and  a  Nunn-McCurdy  breach  in 
September  2008.  At  program  inception,  five  satellites  were  to  be  delivered  at  an 
estimated  cost  of  $5.4  billion.  In  2003  the  program  scope  had  diminished  to  three 
satellites,  but  the  total  program  cost  remained  at  $4.8  billion  (Government  Accountability 
Office,  2003).  This  represents  a  growth  in  average  per  satellite  cost  by  $570  million. 

The  GAO  cites  several  factors  in  the  requirements,  schedule  and  cost  systems  that  led  to 
AEHF‘s  inability  to  meet  its  goals.  The  Department  of  Defense  frequently  altered 
requirements,  pursued  an  overly  aggressive,  unrealistic  schedule,  and  did  not  have  the 
required  funding  to  support  the  activities  and  manpower  to  design  and  build  the  satellites 
faster  (Government  Accountability  Office,  2003). 

Based  on  this  evidence,  it  is  clear  that  the  major  failures  were  primarily  caused  by 
poor  interoperability  in  the  stability  and  realism  characters.  Frequent  requirement  and 
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design  changes  resulted  in  cost  growth  and  schedule  delays.  Although  the  program 
utilized  an  experienced  (incumbent)  contractor  with  strong  technical  maturity,  they 
pursued  an  incredibly  aggressive  schedule  with  inadequate  funding.  This  unrealistic 
schedule  and  cost  effectively  severely  compromised  the  overall  systenTs  ability  to 
accomplish  the  requirements.  The  progranTs  mission  focus  was  also  skewed  in  favor  of 
schedule  and  cost.  Therefore,  the  AEHF  SAIMM  is  scored  as  follows: 

Table  10.  AEHF  SAIMM,  2003 


MISSION  FOCUS 

STABILITY 

DISCIPLINE 

REALISM 

Level  4- 

Accordant 

Mission  success  is  weighted 
significantly  higher  than  cost  and 
schedule;  its  priority  is  reflected  in  risk 
management,  test  planning,  system 
engineering,  and  funding  profiles 

Stable  budgets,  stable  requirements, 
stable  direction,  and  low  personnel 
turnover  are  maintained  across  the 
board;  program  decisions  are  constant; 
technology  is  mature  (TRL7  or  higher) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  prime  contractor)  and 
maintains  strict  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified  and  accomplished  in  a  timely, 
realistic,  and  complete  manner; 
minimal  rework  is  required;  program 
funding  and  management  reserve  is 
realistic  (80/20  cost  confidence  with 
25%  management  reserve) 

Level  3- 

Associated 

Mission  success  is  weighted  higher 
than  cost  and  schedule 

Budgets,  requirements,  direction  and 
personnel  turnover  rarely  change  and 
remain  stable  in  critical  areas;  program 
decisions  rarely  change;  technology  is 
sufficiently  mature  (TRL6) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  major  subcontractor)  and 
maintains  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified;  rework  is  rarely  required; 
program  funding  is  unrealistic 
(between  80/20  and  50/50 cost 
confidence)  with  realistic  management 
reserve  (20%  or  greater) 

Level  2- 

Structured 

Mission  success  is  weighted  equally  to 
cost  and  schedule 

Budgets,  requirements,  direction  or 
personnel  turnover  change  occasionally 
in  a  few  minor  areas;  program  decisions 
change  occasionally;  technology 
requires  maturation  (TRL5) 

The  acquisition  team  has  some 
experience  (previous  experience  on  a 
similar  program  as  a  subcontractor)  and 
utilizes  proven  processes  in  most  areas 

Program  estimates  are  independently 
verified  but  unrealistic  in  a  few  minor 
areas;  rework  is  occasionally  required; 
program  funding  is  unrealistic  (50/50 
cost  confidence)  with  realistic 
management  reserve  (20%  or  greater) 

Level  1  - 

Aligned 

Cost  and  schedule  are  weighted  higher 
than  mission  success 

Budgets,  requirements,  direction  or 
personnel  turnover  change  frequently 
in  several  critical  areas;  program 
decisions  change  frequently; 
technology  requires  significant 
maturation  (TRL4) 

The  acquisition  team  has  limited 
experience  (previous  experience  on  a 
portion  of  a  similar  program)  and  has 
knowledge  of  proven  processes  but 
does  not  use  them  consistently 

Program  estimates  are  unrealistic  in 
several  critical  areas;  rework  is 
frequently  required;  program  funding 
is  unrealistic  with  inadequate  (less 
than  20%)  management  reserve 

Level  0- 
Separated 

Cost  and  schedule  are  weighted 
significantly  higher  than  mission 

success 

Budgets,  requirements,  direction  and 
personnel  turnover  continually  change 
in  many  critical  areas;  program 
decisions  continually  change; 
technology  is  not  mature  (TRL3) 

The  acquisition  team  has  no  experience 
(first-ever  program)  and  does  not  have 
access  to  proven  processes  (they  do  not 
exist) 

Program  estimates  are  unrealistic  and 
focused  on  program  advocacy;  rework 
is  continually  required;  program 
funding  is  unrealistic  with  little  or  no 
management  reserve 

COST 

SCHEDULE 

REQUIREMENTS 


This  SAIMM  score  results  in  the  following  instantiation  of  AEHF: 
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Table  11.  AEHF  Instantiation,  2003 


AEHF  (2003) 

Mission  Focus 

Stability 

Discipline 

Realism 

Cost 

0 

2 

1 

0 

Schedule 

0 

2 

1 

1 

Requirements 

1 

0 

4 

3 

Recall  that  cmax  represents  the  range  of  possible  character  states,  n  signifies  the  number  of 
characters  used  to  instantiate  the  systems,  and  r  is  ordinarily  set  to  2.  Using  Ford‘s 
Sinireai  function  where  r  =  2,  cmax  =  4 ,  and  n  =  4 ,  and  assuming  no  self-interoperability, 
the  interoperability  measurement  yields: 

Table  12.  AEHF  Interoperability  Measurement,  2003 


AEHF  (2003)  -  Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.191 

0.138 

Schedule 

0.191 

0.000 

0.176 

Requirements 

0.138 

0.176 

0.000 

The  AEHF  program  was  again,  albeit  briefly  reviewed  by  the  GAO  roughly  five 
months  before  the  progranTs  first  launch  in  August  of  2010.  The  report  provided  several 
morsels  of  new  evidence  that  can  be  used  to  update  the  program1  s  interoperability 
measurement  from  2003.  The  program  made  significant  progress  since  the  2003  GAO 
report,  but  it  came  at  the  expense  of  hundreds  of  millions  of  dollars  in  cost  growth  and 
multiple  launch  slips  representing  approximately  four  years  of  delay.  The  program 
achieved  strong  stability  and  discipline  in  the  requirements  system,  and  maintained  solid 
maturity  across  the  remaining  systems  and  characters.  The  SAIMM  for  AEHF  at  this 
point  is  as  follows: 
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Table  13.  AEHF  SAIMM,  2010 


MISSION  FOCUS 

STABILITY 

DISCIPLINE 

REALISM 

Level  4- 

Accordant 

Mission  success  is  weighted 
significantly  higher  than  cost  and 
schedule;  its  priority  is  reflected  in  risk 
management,  test  planning,  system 
engineering,  and  funding  profiles 

Stable  budgets,  stable  requirements, 
stable  direction,  and  low  personnel 
turnover  are  maintained  across  the 
board;  program  decisions  are  constant; 
technology  is  mature  (TRL  7  or  higher) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  prime  contractor)  and 
maintains  strict  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified  and  accomplished  in  a  timely, 
realistic,  and  complete  manner; 
minimal  rework  is  required;  program 
funding  and  management  reserve  is 
realistic  (80/20  cost  confidence  with 
25%  management  reserve) 

Level  3- 

Associated 

Mission  success  is  weighted  higher 
than  cost  and  schedule 

Budgets,  requirements,  direction  and 
personnel  turnover  rarely  change  and 
remain  stable  in  critical  areas;  program 
decisions  rarely  change;  technology  is 
sufficiently  mature  (TRL  6) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  major  subcontractor)  and 
maintains  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified;  rework  is  rarely  required; 
program  funding  is  unrealistic 
(between  80/20  and  50/50  cost 
confidence)  with  realistic  management 
reserve  (20%  or  greater) 

Level  2- 

Structured 

Mission  success  is  weighted  equally  to 
cost  and  schedule 

Budgets,  requirements,  direction  or 
personnel  turnover  change  occasionally 
in  a  few  minor  areas;  program  decisions 
change  occasionally;  technology 
requires  maturation  (TRL  5) 

The  acquisition  team  has  some 
experience  (previous  experience  on  a 
similar  program  as  a  subcontractor)  and 
utilizes  proven  processes  in  most  areas 

Program  estimates  are  independently 
verified  but  unrealistic  in  a  few  minor 
areas;  rework  is  occasionally  required; 
program  funding  is  unrealistic  (50/50 
cost  confidence)  with  realistic 
management  reserve  (20%  or  greater) 

Level  1  - 

Aligned 

Cost  and  schedule  are  weighted  higher 
than  mission  success 

Budgets,  requirements,  direction  or 
personnel  turnover  change  frequently 
in  several  critical  areas;  program 
decisions  change  frequently; 
technology  requires  significant 
maturation  (TRL  4) 

The  acquisition  team  has  limited 
experience  (previous  experience  on  a 
portion  of  a  similar  program)  and  has 
knowledge  of  proven  processes  but 
does  not  use  them  consistently 

Program  estimates  are  unrealistic  in 
several  critical  areas;  rework  is 
frequently  required;  program  funding 
is  unrealistic  with  inadequate  (less 
than  20%)  management  reserve 

Level  0- 
Separated 

Cost  and  schedule  are  weighted 
significantly  higher  than  mission 

success 

Budgets,  requirements,  direction  and 
personnel  turnover  continually  change 
in  many  critical  areas;  program 
decisions  continually  change; 
technology  is  not  mature  (TRL  3) 

The  acquisition  team  has  no  experience 
(first-ever  program)  and  does  not  have 
access  to  proven  processes  (they  do  not 
exist) 

Program  estimates  are  unrealistic  and 
focused  on  program  advocacy;  rework 
is  continually  required;  program 
funding  is  unrealistic  with  little  or  no 
management  reserve 

COST 

SCHEDULE 

REQUIREMENTS 

COST,  SCHEDULE  &  REQUIREMENTS 

Which  results  in  the  following  interoperability  instantiation: 

Table  14.  AEHF  Instantiation,  2010 


AEHF  (2010) 

Mission  Focus 

Stability 

Discipline 

Realism 

Cost 

3 

3 

3 

3 

Schedule 

3 

3 

3 

3 

Requirements 

3 

4 

4 

3 

Using  Ford‘s  Simreai  function  where  r  =  2,  cmax  =  4,  and  n  =  4,  and  assuming  no  self¬ 
interoperability,  the  interoperability  measurement  yields: 
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Table  15.  AEHF  Interoperability  Measurement,  2010 


AEHF  (2010)- Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.750 

0.669 

Schedule 

0.750 

0.000 

0.669 

Requirements 

0.669 

0.669 

0.000 

National  Polar-Orbiting  Operational  Environmental  Satellite  (NPOESS) 

NPOESS  was  a  tri-agency  program  (Department  of  Commerce,  National  Air  and 
Space  Administration,  and  the  Department  of  Defense)  that  suffered  through  numerous 
setbacks.  Originally  scheduled  to  launch  in  2006,  the  program  was  restructured  in  2007 
due  to  a  Nunn-McCurdy  breach,  and  its  first  satellite,  known  as  the  NPOESS  Prepatory 
Project,  is  now  scheduled  to  launch  in  late  2011  (Government  Accountability  Office, 
2010).  Earlier  this  year  the  program  was  cancelled  and  split  into  separate  Department  of 
Commerce  and  Department  of  Defense  programs.  The  prime  contractor  for  NPOESS 
was  Northrop  Grumman  Aerospace  Systems  (Los  Angeles,  California)  who  also  had 
significant  experience  building  the  DOD‘s  legacy  weather  satellite,  the  Defense 
Meteorological  Satellite  Program. 

In  2009,  an  independent  review  team  was  commissioned  by  the  NPOESS 
Executive  Committee  to  review  the  program1  s  management  approach  and  to  determine 
key  issues  and  risks.  The  review  team  was  chaired  by  Mr.  Tom  Young,  who  was  also  the 
chair  of  the  2003  Defense  Science  Board  review  team,  and  based  on  their  findings 
(NPOESS  Independent  Review  Team,  2009),  the  NPOESS  program  demonstrated 
significant  issues  across  all  of  the  major  acquisition  systems  as  manifested  heavily  in  the 
mission  focus  and  stability  characters.  Cost  and  schedule  took  priority  over  mission 
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success,  budgets  were  inadequate,  the  management  team  lacked  experience,  and  the 
program‘s  requirements  were  in  conflict  due  to  the  tri-agency  management  structure. 
The  resulting  SAIMM  score: 

Table  16.  NPOESS  SAIMM,  2009 


MISSION  FOCUS 

STABILITY 

DISCIPLINE 

REALISM 

Level  4- 

Accordant 

Mission  success  is  weighted 
significantly  higher  than  cost  and 
schedule;  its  priority  is  reflected  in  risk 
management,  test  planning,  system 
engineering,  and  funding  profiles 

Stable  budgets,  stable  requirements, 
stable  direction,  and  low  personnel 
turnover  are  maintained  across  the 
board;  program  decisions  are  constant; 
technology  is  mature  (TRL7  or  higher) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  prime  contractor)  and 
maintains  strict  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified  and  accomplished  in  a  timely, 
realistic,  and  complete  manner; 
minimal  rework  is  required;  program 
funding  and  management  reserve  is 
realistic  (80/20  cost  confidence  with 
25%  management  reserve) 

Level  3- 

Associated 

Mission  success  is  weighted  higher 
than  cost  and  schedule 

Budgets,  requirements,  direction  and 
personnel  turnover  rarely  change  and 
remain  stable  in  critical  areas;  program 
decisions  rarely  change;  technology  is 
sufficiently  mature  (TRL6) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  major  subcontractor)  and 
maintains  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified;  rework  is  rarely  required; 
program  funding  is  unrealistic 
(between  80/20  and  50/50 cost 
confidence)  with  realistic  management 
reserve  (20%  or  greater) 

Level  2- 

Structured 

Mission  success  is  weighted  equally  to 
cost  and  schedule 

Budgets,  requirements,  direction  or 
personnel  turnover  change  occasionally 
in  a  few  minor  areas;  program  decisions 
change  occasionally;  technology 
requires  maturation  (TRL5) 

The  acquisition  team  has  some 
experience  (previous  experience  on  a 
similar  program  as  a  subcontractor)  and 
utilizes  proven  processes  in  most  areas 

Program  estimates  are  independently 
verified  but  unrealistic  in  a  few  minor 
areas;  rework  is  occasionally  required; 
program  funding  is  unrealistic  (50/50 
cost  confidence)  with  realistic 
management  reserve  (20%  or  greater) 

Level  1  - 

Aligned 

Cost  and  schedule  are  weighted  higher 
than  mission  success 

Budgets,  requirements,  direction  or 
personnel  turnover  change  frequently 
in  several  critical  areas;  program 
decisions  change  frequently; 
technology  requires  significant 
maturation  (TRL4) 

The  acquisition  team  has  limited 
experience  (previous  experience  on  a 
portion  of  a  similar  program)  and  has 
knowledge  of  proven  processes  but 
does  not  use  them  consistently 

Program  estimates  are  unrealistic  in 
several  critical  areas;  rework  is 
frequently  required;  program  funding 
is  unrealistic  with  inadequate  (less 
than  20%)  management  reserve 

Level  0- 
Separated 

Cost  and  schedule  are  weighted 
significantly  higher  than  mission 

success 

Budgets,  requirements,  direction  and 
personnel  turnover  continually  change 
in  many  critical  areas;  program 
decisions  continually  change; 
technology  is  not  mature  (TRL3) 

The  acquisition  team  has  no  experience 
(first-ever  program)  and  does  not  have 
access  to  proven  processes  (they  do  not 
exist) 

Program  estimates  are  unrealistic  and 
focused  on  program  advocacy;  rework 
is  continually  required;  program 
funding  is  unrealistic  with  little  or  no 
management  reserve 

COST 

SCHEDULE 

REQUIREMENTS 

Which  yields  the  following  NPOESS  instantiation: 

Table  17.  NPOESS  Instantiation,  2009 


NPOESS 

Mission  Focus 

Stability 

Discipline 

Realism 

Cost 

0 

2 

1 

2 

Schedule 

1 

2 

1 

1 

Requirements 

1 

0 

2 

1 

Using  Ford‘s  Simrea!  function  where  r  =  2,  cmax  =  4,  and  n  =  4,  and  assuming  no  self¬ 
interoperability,  the  interoperability  measurement  yields: 
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Table  18.  NPOESS  Interoperability  Measurement,  2009 


NPOESS  -  Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.257 

0.188 

Schedule 

0.257 

0.000 

0.203 

Requirements 

0.188 

0.203 

0.000 

Space-Based  Infrared  System  (SBIRS)  High 

In  the  early  1990s,  the  DOD  embraced  a  Total  System  Performance 
Responsibility  (TSPR)  approach  for  system  acquisitions.  In  theory,  TSPR  allowed  the 
government  to  leverage  existing  contractor  management  and  commercial  practices  with 
minimal  oversight  by  turning  government-led  functions  over  to  the  contractor.  In  1996, 
the  SBIRS  program  was  initiated  and  Lockheed  Martin  was  selected  as  the  prime 
contractor  with  Northrup  Grumman  as  their  major  subcontractor.  Northrup  Grumman 
was  both  the  prime  contractor  and  sensor  developer  for  the  legacy  Defense  Support 
Program.  Despite  this  legacy  experience,  the  program  generated  expansive  cost  and 
schedule  failures.  By  the  fall  of  2001,  SBIRS  had  an  estimated  cost  growth  in  excess  of 
$2  billion  (Government  Accountability  Office,  2003).  After  three  Nunn-McCurdy 
violations,  problems  on  the  SBIRS  program  have  persisted  to  the  present  day  as 
manifested  by  a  200%  change  in  total  program  cost  and  a  unit  cost  growth  of  nearly  $2.5 
billion  (Government  Accountability  Office,  2010). 

The  program  instituted  several  independent  reviews  while  the  GAO  continued  to 
monitor  the  program‘s  progress  against  cost,  schedule  and  perfonnance  goals.  Three 
reports  were  issued  in  2003,  2005  and  2008  that  capture  various  findings  from 
independent  reviews,  the  GAO  and  others.  The  causes  for  the  progranfs  cost  growth, 
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schedule  delays  and  performance  issues  are  often  attributed  to  the  TSPR-model,  but  a 
deeper  examination  is  required  to  root  out  the  specific  causes  of  a  -troubled  program  that 
could  be  considered  a  case  study  for  how  not  to  execute  a  space  program”  (Office  of  the 
Undersecretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  2003). 

The  2003  GAO  report  was  issued  after  the  program  restructured  in  2002.  The 
GAO  detennined  that  significant  cost  and  schedule  risk  remained  despite  the 
restructuring.  From  an  acquisition  system  interoperability  perspective,  the  SBIRS  High 
program  had  fatal  flaws  across  all  of  the  interoperability  characters  in  each  system.  At 
this  point  in  time,  the  program  moved  ahead  despite  numerous  best  practice  violations  in 
all  characters.  Requirements  were  not  well  understood  and  the  overall  complexity  of  the 
program  was  underestimated.  The  management  team  was  unable  to  deal  with  this 
untenable  situation,  which  resulted  in  — tluProgram  Office  and  contractor  having  to  spend 
25  of  the  first  60  months  of  the  contract  on  replanning  activities”  (Government 
Accountability  Office,  2003).  The  2003  Defense  Science  Board  summarized  the  core 
issue,  -tn  short,  SBIRS  High  illustrates  that  while  government  and  industry  understand 
how  to  manage  challenging  space  programs,  they  abandoned  fundamentals  and  replaced 
them  with  unproven  approaches  that  promised  significant  savings.  In  so  doing,  they 
accepted  unjustified  risk”  (Office  of  the  Undersecretary  of  Defense  for  Acquisition, 
Technology  and  Logistics,  2003).  The  following  SAIMM  results: 
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Table  19.  SBIRS  SAIMM,  2003 


MISSION  FOCUS 

STABILITY 

DISCIPLINE 

REALISM 

Level  4- 

Accordant 

Mission  success  is  weighted 
significantly  higher  than  cost  and 
schedule;  its  priority  is  reflected  in  risk 
management,  test  planning,  system 
engineering,  and  funding  profiles 

Stable  budgets,  stable  requirements, 
stable  direction,  and  low  personnel 
turnover  are  maintained  across  the 
board;  program  decisions  are  constant; 
technology  is  mature  (TRL7  or  higher) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  prime  contractor)  and 
maintains  strict  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified  and  accomplished  in  a  timely, 
realistic,  and  complete  manner; 
minimal  rework  is  required;  program 
funding  and  management  reserve  is 
realistic  (80/20  cost  confidence  with 
25%  management  reserve) 

Level  3- 

Associated 

Mission  success  is  weighted  higher 
than  cost  and  schedule 

Budgets,  requirements,  direction  and 
personnel  turnover  rarely  change  and 
remain  stable  in  critical  areas;  program 
decisions  rarely  change;  technology  is 
sufficiently  mature  (TRL6) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  major  subcontractor)  and 
maintains  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified;  rework  is  rarely  required; 
program  funding  is  unrealistic 
(between  80/20  and  50/50 cost 
confidence)  with  realistic  management 
reserve  (20%  or  greater) 

Level  2- 

Structured 

Mission  success  is  weighted  equally  to 
cost  and  schedule 

Budgets,  requirements,  direction  or 
personnel  turnover  change  occasionally 
in  a  few  minor  areas;  program  decisions 
change  occasionally;  technology 
requires  maturation  (TRL5) 

The  acquisition  team  has  some 
experience  (previous  experience  on  a 
similar  program  as  a  subcontractor)  and 
utilizes  proven  processes  in  most  areas 

Program  estimates  are  independently 
verified  but  unrealistic  in  a  few  minor 
areas;  rework  is  occasionally  required; 
program  funding  is  unrealistic  (50/50 
cost  confidence)  with  realistic 
management  reserve  (20%  or  greater) 

Level  1  - 

Aligned 

Cost  and  schedule  are  weighted  higher 
than  mission  success 

Budgets,  requirements,  direction  or 
personnel  turnover  change  frequently 
in  several  critical  areas;  program 
decisions  change  frequently; 
technology  requires  significant 
maturation  (TRL4) 

The  acquisition  team  has  limited 
experience  (previous  experience  on  a 
portion  of  a  similar  program)  and  has 
knowledge  of  proven  processes  but 
does  not  use  them  consistently 

Program  estimates  are  unrealistic  in 
several  critical  areas;  rework  is 
frequently  required;  program  funding 
is  unrealistic  with  inadequate  (less 
than  20%)  management  reserve 

Level  0- 
Separated 

Cost  and  schedule  are  weighted 
significantly  higher  than  mission 

success 

Budgets,  requirements,  direction  and 
personnel  turnover  continually  change 
in  many  critical  areas;  program 
decisions  continually  change; 
technology  is  not  mature  (TRL3) 

The  acquisition  team  has  no  experience 
(first-ever  program)  and  does  not  have 
access  to  proven  processes  (they  do  not 
exist) 

Program  estimates  are  unrealistic  and 
focused  on  program  advocacy;  rework 
is  continually  required;  program 
funding  is  unrealistic  with  little  or  no 
management  reserve 

COST 

SCHEDULE 

REQUIREMENTS 

Which  yields  the  interoperability  instantiation  of: 

Table  20.  SBIRS  Instantiation,  2003 


SBIRS  (2003) 

Mission  Focus 

Stability 

Discipline 

Realism 

Cost 

1 

1 

1 

0 

Schedule 

0 

1 

1 

1 

Requirements 

1 

0 

0 

0 

Using  Ford‘s  Simreai  function  where  r  =  2,  cmax  =  4,  and  n  =  4,  and  assuming  no  self¬ 
interoperability,  the  interoperability  measurement  yields: 
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Table  21.  SBIRS  Interoperability  Measurement,  2003 


SBIRS  (2003)-  Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.154 

0.103 

Schedule 

0.154 

0.000 

0.094 

Requirements 

0.103 

0.094 

0.000 

In  2005,  many  of  the  GAO‘s  concerns  were  realized  as  the  program  continued  to 
experience  difficulties.  During  this  period,  the  Secretary  of  Defense  was  directed  by 
Congress  to  provide  a  report  -en  the  cause  of  the  most  recent  SBIRS  cost  increases, 
schedule  delays,  and  technical  problems;  the  most  recent  Defense  Support  Program  gap 
analysis  and  any  effect  that  further  delays  will  have  on  U.S.  early  warning,  technical 
intelligence,  and  missile  defense  capabilities;  steps  taken  to  address  the  most  recent 
SBIRS  technical  difficulties;  any  adjustments  in  management  and  contract  arrangements 
with  the  contractor  to  reflect  the  most  recent  program  challenges;  remaining  risk  areas; 
and  an  assessment  of  the  confidence  level  in  the  SBIRS  schedule  and  cost  estimates 
current  as  of  October  1,  2004”  (Office  of  the  Secretary  of  Defense,  2005).  The 
subsequent  report  cited  a  lack  of  -sound  system  engineering  processes  and 
procedures... insufficient  schedule  and  budget... process  escapes”  and  -an  inadequate 
architecture  design  and  a  flawed  flight  software  development  plan”  (Office  of  the 
Secretary  of  Defense,  2005).  The  report  also  cites  several  program  initiatives  designed  to 
-better  focus  resources  on  key  program  issues,”  and  to  — drmally  adopt  an  =event-driven‘ 
approach,  replacing  the  =schedule-driven‘  mentality  of  the  past. .  .ensures  the  program  no 
longer  enters  an  activity  unless  the  probability  for  success  is  high”  (Office  of  the 
Secretary  of  Defense,  2005).  Adjustments  to  the  progranTs  business  rhythm,  testing 
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activities,  and  contract  were  also  noted.  These  findings  do  not  represent  a  major 
improvement  in  the  SBIRS  program  however  as  the  findings  from  2003  were  either 
realized  or  persisted.  As  such,  there  is  not  enough  evidence  presented  in  the  2005  report 
to  significantly  alter  the  interoperability  measurement  from  2003. 

Fast  forward  to  2008;  SBIRS  has  just  experienced  a  major  setback  with  its  flight 
software  and  the  GAO  once  again  warns  against  an  -ambitious,”  even  -optimistic”  plan 
to  resolve  the  problems.  The  2008  GAO  report  found  that  the  issues  largely  stemmed 
from  the  program1  s  test  and  evaluation  area,  however  it  is  more  important  to  examine  the 
GAO‘s  assessment  of  the  progranTs  plan  going  forward.  With  respect  to  the  flight 
software  issue,  significant  interoperability  issues  remained  in  the  mission  focus,  discipline 
and  realism  characters.  Cost  and  schedule  estimates  were  overly  optimistic,  and  there 
was  little  to  no  margin  to  react  to  more  problems  if  they  occurred.  The  program  also 
requested  waivers  to  forgo  -disciplined  processes,”  which  marginalized  mission 
assurance  (Government  Accountability  Office,  2008). 

Just  prior  to  the  GAO  report,  in  2007,  another  independent  review  team  was 
commissioned  to  review  the  progranTs  status.  The  following  chart  summarizes  the 
progranTs  progress  against  this  independent  review  teanTs  findings  and 
recommendations: 
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Table  22.  Independent  Program  Assessment  (Government  Accountability  Office, 


2008) 


Table  2:  IPA  Findings,  Recommendations,  and  Status  of  Implementation 

Finding 

Recommendation 

Implemented? 

(as  of  April  2008) 

1.  Lockheed  Martin’s  program  process 
discipline  is  poor 

•  Engage  Lockheed  Martin  functional  areas  and  ensure  that 
processes  are  being  followed 

Yes 

2.  Air  Force  has  limited  management  control 
over  SBIRS 

•  Amend  contract  to  provide  necessary  management  control 

Yes 

3.  Adversarial  relationships  exist  between  Air 
Force  and  Lockheed  Martin 

•  Fix  responsibility,  accountability,  and  authority  disconnects 

Yes 

4.  Government  organizational  structure  is 
flawed  because  cost  and  schedule 
responsibilities  are  separated. 

•  Combine  in  a  single  office  the  review  of  contractor  cost  and 
schedule  data 

Yes 

5.  Focal  point  for  FSS  completion  is  needed 

•  Designate  a  program  manager  within  flight  software  system 

•  Establish  giver/receiver  relationships 

Yes 

Source:  Aerospace  Corporation  (data)  and  U.S.  Air  Force  (data):  GAO  (analysis  and  presentation). 


This  table  indicates  the  program  has  improved  interoperability  between  the  cost  and 
schedule  systems,  as  well  as  between  the  government  and  contractor  team.  Based  on  the 
2007  independent  review  teanTs  findings  and  the  2008  GAO  report,  there  is  sufficient 
evidence  to  update  the  program1  s  SAIMM  and  interoperability  measurements. 
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Table  23.  SBIRS  SAIMM,  2008 


MISSION  FOCUS 

STABILITY 

DISCIPLINE 

REALISM 

Level  4- 

Accordant 

Mission  success  is  weighted 
significantly  higher  than  cost  and 
schedule;  its  priority  is  reflected  in  risk 
management,  test  planning,  system 
engineering,  and  funding  profiles 

Stable  budgets,  stable  requirements, 
stable  direction,  and  low  personnel 
turnover  are  maintained  across  the 
board;  program  decisions  are  constant; 
technology  is  mature  (TRL7  or  higher) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  prime  contractor)  and 
maintains  strict  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified  and  accomplished  in  a  timely, 
realistic,  and  complete  manner; 
minimal  rework  is  required;  program 
funding  and  management  reserve  is 
realistic  (80/20  cost  confidence  with 
25%  management  reserve) 

Level  3- 

Associated 

Mission  success  is  weighted  higher 
than  cost  and  schedule 

Budgets,  requirements,  direction  and 
personnel  turnover  rarely  change  and 
remain  stable  in  critical  areas;  program 
decisions  rarely  change;  technology  is 
sufficiently  mature  (TRL6) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  major  subcontractor)  and 
maintains  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified;  rework  is  rarely  required; 
program  funding  is  unrealistic 
(between  80/20  and  50/50 cost 
confidence)  with  realistic  management 
reserve  (20%  or  greater) 

Level  2- 

Structured 

Mission  success  is  weighted  equally  to 
cost  and  schedule 

Budgets,  requirements,  direction  or 
personnel  turnover  change  occasionally 
in  a  few  minor  areas;  program  decisions 
change  occasionally;  technology 
requires  maturation  (TRL5) 

The  acquisition  team  has  some 
experience  (previous  experience  on  a 
similar  program  as  a  subcontractor)  and 
utilizes  proven  processes  in  most  areas 

Program  estimates  are  independently 
verified  but  unrealistic  in  a  few  minor 
areas;  rework  is  occasionally  required; 
program  funding  is  unrealistic  (50/50 
cost  confidence)  with  realistic 
management  reserve  (20%  or  greater) 

Level  1  - 

Aligned 

Cost  and  schedule  are  weighted  higher 
than  mission  success 

Budgets,  requirements,  direction  or 
personnel  turnover  change  frequently 
in  several  critical  areas;  program 
decisions  change  frequently; 
technology  requires  significant 
maturation  (TRL4) 

The  acquisition  team  has  limited 
experience  (previous  experience  on  a 
portion  of  a  similar  program)  and  has 
knowledge  of  proven  processes  but 
does  not  use  them  consistently 

Program  estimates  are  unrealistic  in 
several  critical  areas;  rework  is 
frequently  required;  program  funding 
is  unrealistic  with  inadequate  (less 
than  20%)  management  reserve 

Level  0- 
Separated 

Cost  and  schedule  are  weighted 
significantly  higher  than  mission 

success 

Budgets,  requirements,  direction  and 
personnel  turnover  continually  change 
in  many  critical  areas;  program 
decisions  continually  change; 
technology  is  not  mature  (TRL3) 

The  acquisition  team  has  no  experience 
(first-ever  program)  and  does  not  have 
access  to  proven  processes  (they  do  not 
exist) 

Program  estimates  are  unrealistic  and 
focused  on  program  advocacy;  rework 
is  continually  required;  program 
funding  is  unrealistic  with  little  or  no 
management  reserve 

COST 

SCHEDULE 

REQUIREMENTS 

Which  yields  the  following  instantiation: 

Table  24.  SBIRS  Instantiation,  2008 


SBIRS  (2008) 

Mission  Focus 

Stability 

Discipline 

Realism 

Cost 

1 

2 

3 

2 

Schedule 

0 

2 

3 

2 

Requirements 

1 

3 

2 

1 

Using  Ford‘s  Simreai  function  where  r  =  2,  cmax  =  4,  and  n  =  4,  and  assuming  no  self¬ 
interoperability,  the  interoperability  measurement  yields: 
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Table  25.  SBIRS  Interoperability  Measurement,  2008 


SBIRS  (2008)  -  Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.410 

0.367 

Schedule 

0.410 

0.000 

0.328 

Requirements 

0.367 

0.328 

0.000 

The  GAO  provided  a  brief  assessment  of  the  SBIRS  High  program  again  in  2010 
and  noted  that  — Id  three  critical  technologies. .  .are  now  mature”  and  -99  percent  of 
the... expected  design  drawings  are  now  releasable,”  but  also  cited  -design-related 
problems”  and  an  assessment  of  cost  and  schedule  as  — Igh  risk”  (Government 
Accountability  Office,  2010).  This  update  delivers  an  adequate  amount  of  new 
information  to  update  the  program‘s  SAIMM.  The  program  has  matured  significantly  in 
the  mission  focus  character,  but  continues  to  suffer  from  a  lack  of  realism.  Stability  has 
improved  to  a  lesser  degree  as  the  maturity  of  the  design  and  requirements  is  high,  but 
discipline  remains  unchanged  due  to  continued  problems  and  unknowns  with  the  flight 
software,  and  an  apparent  inability  to  control  and  plan  the  remaining  work  required.  The 
2010  GAO  assessment  yields  the  following  SAIMM: 
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Table  26.  SBIRS  SAIMM,  2010 


MISSION  FOCUS 

STABILITY 

DISCIPLINE 

REALISM 

Level  4- 

Accordant 

Mission  success  is  weighted 
significantly  higher  than  cost  and 
schedule;  its  priority  is  reflected  in  risk 
management,  test  planning,  system 
engineering,  and  funding  profiles 

Stable  budgets,  stable  requirements, 
stable  direction,  and  low  personnel 
turnover  are  maintained  across  the 
board;  program  decisions  are  constant; 
technology  is  mature  (TRL  7  or  higher) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  prime  contractor)  and 
maintains  strict  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified  and  accomplished  in  a  timely, 
realistic,  and  complete  manner; 
minimal  rework  is  required;  program 
funding  and  management  reserve  is 
realistic  (80/20  cost  confidence  with 
25%  management  reserve) 

Level  3- 

Associated 

Mission  success  is  weighted  higher 
than  cost  and  schedule 

Budgets,  requirements,  direction  and 
personnel  turnover  rarely  change  and 
remain  stable  in  critical  areas;  program 
decisions  rarely  change;  technology  is 
sufficiently  mature  (TRL  6) 

The  acquisition  team  is  experienced 
(previous  experience  on  the  legacy 
program  as  a  major  subcontractor)  and 
maintains  adherence  to  proven 
programmatic  and  system  engineering 
processes 

Program  estimates  are  independently 
verified;  rework  is  rarely  required; 
program  funding  is  unrealistic 
(between  80/20  and  50/50  cost 
confidence)  with  realistic  management 
reserve  (20%  or  greater) 

Level  2- 

Structured 

Mission  success  is  weighted  equally  to 
cost  and  schedule 

Budgets,  requirements,  direction  or 
personnel  turnover  change  occasionally 
in  a  few  minor  areas;  program  decisions 
change  occasionally;  technology 
requires  maturation  (TRL  5) 

The  acquisition  team  has  some 
experience  (previous  experience  on  a 
similar  program  as  a  subcontractor)  and 
utilizes  proven  processes  in  most  areas 

Program  estimates  are  independently 
verified  but  unrealistic  in  a  few  minor 
areas;  rework  is  occasionally  required; 
program  funding  is  unrealistic  (50/50 
cost  confidence)  with  realistic 
management  reserve  (20%  or  greater) 

Level  1  - 

Aligned 

Cost  and  schedule  are  weighted  higher 
than  mission  success 

Budgets,  requirements,  direction  or 
personnel  turnover  change  frequently 
in  several  critical  areas;  program 
decisions  change  frequently; 
technology  requires  significant 
maturation  (TRL  4) 

The  acquisition  team  has  limited 
experience  (previous  experience  on  a 
portion  of  a  similar  program)  and  has 
knowledge  of  proven  processes  but 
does  not  use  them  consistently 

Program  estimates  are  unrealistic  in 
several  critical  areas;  rework  is 
frequently  required;  program  funding 
is  unrealistic  with  inadequate  (less 
than  20%)  management  reserve 

Level  0- 
Separated 

Cost  and  schedule  are  weighted 
significantly  higher  than  mission 

success 

Budgets,  requirements,  direction  and 
personnel  turnover  continually  change 
in  many  critical  areas;  program 
decisions  continually  change; 
technology  is  not  mature  (TRL  3) 

The  acquisition  team  has  no  experience 
(first-ever  program)  and  does  not  have 
access  to  proven  processes  (they  do  not 
exist) 

Program  estimates  are  unrealistic  and 
focused  on  program  advocacy;  rework 
is  continually  required;  program 
funding  is  unrealistic  with  little  or  no 
management  reserve 

COST 

SCHEDULE 

REQUIREMENTS 

COST,  SCHEDULE  &  REQUIREMENTS 

Resulting  in  the  following  interoperability  instantiation: 

Table  27.  SBIRS  Instantiation,  2010 


SBIRS  (2010) 

Mission  Focus 

Stability 

Discipline 

Realism 

Cost 

3 

3 

3 

1 

Schedule 

3 

3 

3 

1 

Requirements 

3 

4 

2 

1 

Using  Ford‘s  Simreai  function  where  r  =  2,  cmax  =  4,  and  n  =  4,  and  assuming  no  self¬ 
interoperability,  the  interoperability  measurement  yields: 
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Table  28.  SBIRS  Interoperability  Measurement,  2010 


SBIRS  (2010)-  Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.625 

0.515 

Schedule 

0.625 

0.000 

0.515 

Requirements 

0.515 

0.515 

0.000 

Interpreting  the  Results 

The  interoperability  measurements  for  AEHF,  NPOESS  and  SBIRS  High 
demonstrate  how  the  SAIMM  can  be  used  to  measure  a  space  acquisition  program1  s 
interoperability  maturity,  and  then  measure  interoperability  between  its  major  systems. 
The  measurements  also  show  how  the  interoperability  can  change  over  time  and  with 
respect  to  program  phases.  A  specific  tie  between  the  interoperability  measurement  and 
actual  cost,  schedule  and  requirements  performance  has  not  been  directly  established,  but 
is  indirectly  captured  by  using  the  examples  of  the  past.  Therefore,  it  is  reasonable  to 
assume  that  a  measurement  of  a  failed  program1  s  interoperability  can  provide  a  baseline 
from  which  to  compare  other  programs1  interoperability  and  chance  of  success  or  failure. 

For  AEHF,  interoperability  measurements  were  provided  in  2003  and  2010. 
According  to  GAO  reports,  during  that  time  the  program1  s  overall  cost  nearly  doubled 
from  4.8  to  10.4  billion  dollars  and  the  predicted  launch  date  slipped  approximately  four 
years.  The  interoperability  measurements  also  exhibit  dramatic  differences: 

Table  29.  AEHF  Interoperability  Measurement,  2003 


AEHF  (2003)  -  Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.191 

0.138 

Schedule 

0.191 

0.000 

0.176 

Requirements 

0.138 

0.176 

0.000 
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Table  30.  AEHF  Interoperability  Measurement,  2010 


AEHF  (2010)-  Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.750 

0.669 

Schedule 

0.750 

0.000 

0.669 

Requirements 

0.669 

0.669 

0.000 

The  program  launched  shortly  after  the  2010  interoperability  measurement  was  taken.  It 
is  expected  that  a  program1  s  interoperability  measurement  will  converge  to  1  as  it 
approaches  a  launch  since  most  of  the  mandatory  reviews  and  maturity  gates  for  the 
program  will  have  been  passed.  Accordingly,  an  increase  in  the  program1  s  measure  of 
interoperability  indicates  an  increase  the  program1  s  mission  readiness. 

The  NPOESS  program  experienced  extreme  difficulties  and  was  eventually 
cancelled  as  a  result.  According  to  GAO  records,  the  program  cost  estimate  was  6.1 
billion  dollars  in  2003,  but  ballooned  to  13.1  billion  dollars  in  2010.  During  this  same 
period,  the  predicted  launch  date  moved  from  2009  to  2014.  The  2003  NPOESS 
interoperability  scores  are  reasonably  close  to  the  2003  AEHF  scores,  and  point  toward 
significant  program  issues  and  a  severely  hampered  ability  to  achieve  cost,  schedule  and 
perfonnance  goals. 

Table  31.  NPOESS  Interoperability  Measurement,  2009 


NPOESS- Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.257 

0.188 

Schedule 

0.257 

0.000 

0.203 

Requirements 

0.188 

0.203 

0.000 

The  SBIRS  High  program  is  a  notorious  example  of  inability  to  meet  cost, 
schedule  and  performance  goals.  Between  2003  and  2010,  the  progranTs  cost  grew  by 
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5.1  billion  dollars  and  the  launch  date  slipped  approximately  four  years.  Unit  cost  grew 
by  almost  two  billion  dollars.  The  progranTs  interoperability  scores  were  lower  than 
both  the  AEHF  and  NPOESS  scores  in  2003  due  to  severe  problems  across  all 
interoperability  characters: 

Table  32.  SBIRS  Interoperability  Measurement,  2003 


SBIRS  (2003)-  Sim  Rea, 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.154 

0.103 

Schedule 

0.154 

0.000 

0.094 

Requirements 

0.103 

0.094 

0.000 

In  2008,  the  program  improved  in  three  out  of  four  areas,  but  continued  to  place  mission 
success  at  risk  due  to  cost  and  schedule  goals  and  constraints.  The  program  was  readying 
for  launch,  but  the  interoperability  scores  did  not  exhibit  a  level  of  interoperability 
maturity  comparable  to  AEHF  just  prior  to  its  launch  in  2010: 

Table  33.  SBIRS  Interoperability  Measurement,  2008 


SBIRS  (2008)-  Sim  Real 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.410 

0.367 

Schedule 

0.410 

0.000 

0.328 

Requirements 

0.367 

0.328 

0.000 

This  difference  in  interoperability  maturity  reflects  the  substantial  risks  and  issues 
remaining  on  the  program.  In  2010,  the  program  continued  to  experience  problems  due 
to  flight  software  issues.  The  GAO  and  other  independent  estimates  stressed  a  lack  of 
realism  in  the  program1  s  ability  and  plan  to  resolve  the  problem.  Despite  a  marked 
increase  in  the  mission  focus,  stability  and  discipline  characters,  the  program  remained 
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less  interoperable  in  the  realism  character.  The  interoperability  scores  again  indicated 
improvement,  but  did  not  reach  the  same  level  as  AEHF  just  prior  to  their  launch: 

Table  34.  SBIRS  Interoperability  Measurement,  2010 


SBIRS  (2010)-  Sim  Rea, 

Cost 

Schedule 

Requirements 

Cost 

0.000 

0.625 

0.515 

Schedule 

0.625 

0.000 

0.515 

Requirements 

0.515 

0.515 

0.000 

The  SBIRS  High  program  now  projects  a  launch  date  in  early  2011. 

These  high-level  correlations  between  a  program1  s  interoperability  score  and  its 
resulting  performance  reveal  the  interoperability  measurement4  s  ability  to  serve  as  a 
leading  indicator  for  space  acquisition  success  or  failure.  The  measurement  also  provides 
insight  into  where  interoperability  can  be  improved  to  increase  chances  of  program 
success. 

Summary 

A  method  to  measure  space  acquisition  interoperation,  using  system 
interoperability,  was  developed  based  upon  Ford‘s  interoperability  measurement  work. 
The  measurement  was  facilitated  by  a  maturity  model  that  was  built  using  the  lessons 
learned  and  best  practices  from  space  acquisition  failures  of  the  past.  The  method 
measured  interoperability  between  space  acquisition  systems,  S  =  {cost,  schedule, 
requirements},  using  key  and  driving  acquisition  characters,  X  =  {mission  focus,  stability, 
discipline,  realism}  based  on  five  character  state  levels,  C  =  {separated,  aligned, 
structured,  associated,  accordant},  and  applied  to  three  specific  acquisition  programs. 


The  resulting  measurements  were  then  compared  to  demonstrate  how  the  method  can  be 
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used  as  a  leading  indicator  of  the  program‘s  ability  to  execute  the  required  level  of 
perfonnance  on  time  and  within  budget. 


66 


V.  Conclusions  and  Recommendations 


...a  host  of problems  not  previously  viewed  as  interoperability  related  can  now  be  looked  at 
as  such.  This  means  that  many  old  problems  can  be  solved  in  a  new  way,  possibly  lending 
insight  or  providing  a  means  of  reporting  progress  not  previously  available. 

— Thomas  C.  Ford  (Ford,  2008) 

This  is  a  logical  and  progressive  evolution  in  warfare,  yet  its  tenets  remain 
undemonstrated  and  unproven... The  network-centric  warfare  objective  needs  further 

investigation  and  technological  exploitation  for  it  to  be...  a  workable  system. 

— Lt.  Col.  Edmund  C.  Blash,  USAR  (Blash,  2003) 

Conclusions  of  Research 

Ford‘s  interoperability  measurement  using  a  maturity  model  is  indeed  a  powerful 
tool  that  can  be  used  to  measure  the  quality  of  acquisition  system  interactions.  As  Ford 
notes,  SimReai  has  the  capability  of  yielding  very  precise  similarity  measures  of  system 
instantiations  limited  only  by  the  number  of  characters  and  the  precision  of  those 
characters1  states”  (Ford,  2008).  In  this  case,  the  maturity  model  dictates  the  number  and 
precision  of  characters.  The  SAIMM  was  built  upon  experts1  analysis  of  past  space 
acquisition  failures  and  their  recommendations  for  improvement.  In  other  words,  the 
SAIMM  captures  the  specific  system  attributes  that  must  be  present  and  mature  in  order 
for  a  program  to  avoid  the  mistakes  of  past  space  acquisition  programs.  These  attributes 
are  universal  and  may  be  applied  to  any  major  space  acquisition  program,  but  the  model4 s 
precision  suffers  as  a  result.  As  such,  a  SAIMM  facilitated  interoperability  measurement 
is  best  used  on  a  macro  level  to  gage  a  program4 s  health  and  status. 

The  SAIMM  is  principally  based  upon  the  lessons  learned  from  space  acquisition 
program  failures,  and  does  not  consider  expert  analysis  of  programs  that  were  deemed 
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successful.  Finding  examples  of  program  successes  is  extremely  difficult  due  to  the 
extent  and  complexity  of  space  acquisition  programs,  and  little,  if  any  expert  analyses 
exist  in  the  publically  available  literature.  The  SAIMM  is  valuable  because  it  is  based  on 
the  cause  and  effect  of  real-world  events,  but  it  is  important  to  note  that  SAIMM  is  rooted 
in  lessons  learned  almost  exclusively  from  programs  that  failed  to  meet  their  goals. 

Future  versions  of  the  SAIMM  or  similar  models  must  strive  to  capture  and  incorporate 
evidence  from  program  successes  to  improve  the  robustness  and  richness  of  the  model. 

In  conclusion,  the  concept  of  acquisition  interoperability  was  described  and  linked 
to  program  success  or  failure  by  utilizing  a  maturity  model  based  upon  past  space 
acquisition  experience.  The  method  was  demonstrated  by  measuring  the  interoperability 
between  the  cost,  schedule  and  requirement  systems  in  three  major  space  programs 
during  various  acquisition  phases.  The  results  were  compared  to  validate  the  utility  of 
the  method  as  a  potential  leading  indicator  of  space  acquisition  success  or  failure.  LTC 
BlaslTs  caveat  must  be  heeded  however;  further  investigation”  is  indeed  required  to 
mature  the  model  and  understand  the  relationship  between  interoperability  and  program 
perfonnance  before  it  can  be  used  as  an  effective  space  acquisition  metric. 

Recommendations  for  Future  Research 

Ford  states  that  —Flic  flexibility  of  the  method  supports  the  instantiation  of 
systems  at  any  level  of  abstraction,  with  resultant  interoperability  measurements  at  any 
desired  level  of  precision.”  The  SAIMM  and  resulting  interoperability  measurements  are 
captured  at  a  very  high  level  of  abstraction  and  should  only  be  used  at  the  macro  level. 
Future  research  is  needed  to  increase  the  depth  and  breadth  of  the  SAIMM  in  order  to 
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enable  a  more  precise  measurement.  As  alluded  to  earlier,  the  bulk  of  the  publically 
available  knowledge  on  space  acquisition  focuses  on  programs  that  have  or  are  currently 
not  meeting  their  cost,  schedule  and  performance  targets.  Very  few  analyses  of 
-successful”  space  acquisition  programs  have  been  conducted,  and  the  addition  of  lessons 
learned  based  on  successful  programs  will  improve  the  SAIMM‘s  ability  to  properly 
characterize  a  space  program‘s  acquisition  interoperability.  Additionally,  the  bulk  of  the 
information  used  to  build  the  SAIMM  was  supplied  by  the  GAO  and  independent  review 
teams  led  by  Mr.  Tom  Young.  Although  extremely  valuable,  there  are  other  sources  of 
data  that  can  be  used  to  improve  the  SAIMM ‘s  precision.  This  may  limit  the  ability  of 
the  material  to  be  publically  released,  but  would  still  provide  benefit  to  most  government 
entities. 

Further  research  must  also  consider  the  fundamental  building  blocks  of  SAIMM 
and  the  interoperability  measurements  used  in  this  thesis.  Additional  systems  and 
characters,  to  include  the  way  they  are  captured  in  the  model  itself,  can  mature  the 
SAIMM  for  both  generic  and  specific  purposes.  Further  decomposition  of  the  cost, 
schedule  and  requirements  systems  could  improve  the  precision  of  the  SAIMM.  For 
example,  the  requirements  system  could  be  broken  into  the  requirements  (user-centric) 
and  design  (program  office-centric)  systems.  The  purpose  and  context  of  the 
measurement,  as  well  as  increased  knowledge  of  lessons  learned  will  reveal  new  and 
important  characters  for  consideration.  The  interoperation  between  the  cost,  schedule 
and  requirements  systems  processes,  methods  and  tools  should  be  considered,  as  well  as 
the  influence  of  the  systems1  interoperation  in  the  flexibility  and  transparency  characters. 
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It  is  also  recommended  that  future  research  further  leverage  the  benefit  of 
hindsight  to  solidify  and  better  quantify  the  relationship  between  acquisition  system 
interoperability  and  a  program‘s  cost,  schedule  and  performance.  One  possible  way  to  do 
this  would  be  to  build  a  baseline  of  interoperability  measurements  across  many  space 
acquisition  programs  during  various  acquisition  phases.  This  baseline  of  interoperability 
measurements  could  then  be  compared  to  the  actual  cost,  schedule  and  mission 
perfonnance  of  the  programs  to  infer  a  cost  per  unit  of  interoperability  metric.  A  simple 
example  might  consider  how  much  a  program4 s  interoperability  improved  or  declined 
between  systems,  and  how  much  cost,  schedule  or  perfonnance  was  impacted  as  a  result. 
This  effort  would  likely  require  extensive  analysis  and  a  more  precise  SAIMM  in  order  to 
produce  meaningful  results. 

An  analysis  of  how  the  program4  s  acquisition  phase  and  context  affects  the 

interoperability  measurement  and  program  performance  would  also  serve  as  a  valuable 

contribution  to  this  area  of  research.  Ford  states  (Ford,  2008): 

Interoperability  is  generally  time  variant.  For  example,  atmospheric  effects  due  to 
the  changes  from  night  to  day  will  degrade  the  optical  interoperability  of 
reconnaissance  satellites  and  ground  targets.  Similarly,  the  directional 
interoperability  of  an  attacker  and  his  target  may  increase  as  the  attacker  has 
ingressed  long  enough  to  come  in  range  of  the  target.  Finally,  end-to-end 
computer  interoperability  may  improve  or  diminish  with  changes  in  network 
congestion  tied  to  worker  shift  changes,  lunchtime  usage,  etc. 

Within  the  context  of  space  acquisitions,  it  is  important  to  consider  how  the 

interoperability  characters  vary  in  time  based  on  the  program's  acquisition  phase.  For 

instance,  the  character  stability  may  prove  to  be  more  valuable  later  in  a  program's  life 

than  earlier.  It  should  also  be  recognized  that  the  characters  selected  for  the  SAIMM  may 
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not  adequately  capture  the  interoperability  needs  across  the  program‘s  acquisition 
lifecycle.  Additional  characters  may  need  to  be  developed  to  fully  portray  the  key  and 
driving  interoperability  factors  during  each  phase  of  the  acquisition  program. 

The  concept  above  could  be  expanded  to  include  statistical  predictions  based 
upon  the  baseline  of  past  space  acquisition  experience.  Distribution  models  or  Monte 
Carlo  analysis  could  be  used  to  correlate  interoperability  measurements  against  actual 
program  perfonnance.  A  series  of  probability  curves  or  other  tools  could  then  link  a 
measurement  of  a  current  program1  s  acquisition  system  interoperability  to  a  probability 
of  success  or  failure,  i.e.  program  x‘s  interoperability  score  of  y  between  the  cost  and 
schedule  systems  indicates  the  program  has  a  65%  chance  of  exceeding  its  current 
budget. 

Finally,  it  is  recommended  that  prospective  research  consider  how  the  acquisition 
interoperability  measurement  be  incorporated  into  a  larger  framework  of  metrics,  most 
likely  driven  by  the  -Systems  Engineering  Leading  Indicators  Guide”  referred  to  earlier. 

Summary 

This  research  combines  the  power  of  Ford‘s  interoperability  measurement  with 
the  lessons  harvested  from  decades  of  space  acquisition  experience,  and  as  such,  is 
inherently  tied  to  overall  acquisition  effectiveness.  It  provides  a  way  to  characterize  a 
progranTs  health  by  measuring  its  ability  to  interoperate  between  the  cost,  schedule  and 
requirements  systems.  The  measurement  can  be  used  as  a  leading  indicator  of  program 
perfonnance,  a  guide  to  locate  and  solve  issues,  and  can  be  tailored  for  use  in  many 
different  contexts  and  situations. 
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There  appears  to  be  great  utility  and  power  in  this  measurement,  but  much  work 
remains  to  be  done  in  order  to  operationalize  it.  The  institutions  responsible  for  space 
program  execution,  and  more  importantly,  delivering  capability  to  the  warfighter  and  our 
nation,  must  continue  to  foster  and  invest  in  ways  to  measure  and  improve  programmatic 
and  system  engineering  rigor. 
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Appendix  A  -  Independent  Assessments  of  AEHF,  NPOESS  and  SBIRS  High 

In  2003,  the  GAO  assessed  the  AEHF  program‘s  status  as  follows  (Government 
Accountability  Office,  2003): 

2003  GAO  reported  in  the  early  phases  of  the  program,  DOD  substantially  and 
frequently  altered  its  requirements;  the  system  design  changed.  While  considered 
necessary,  some  changes  increased  costs  by  hundred  of  millions  of  dollars  and 
caused  scheduling  delays. 

2003  GAO  reported  that  once  DOD  decided  to  accelerate  its  plans  to  build  the 
satellites,  the  contractors  proposed  and  DOD  agreed  to  support  a  high-risk 
schedule  that  turned  out  to  be  overly  optimistic  and  highly  compressed — leaving 
little  room  for  error  and  depending  on  a  chain  of  events  taking  place  at  certain 
times.  Substantial  delays  occurred  when  some  events,  such  as  the  award  of  the 
contract  or  the  availability  of  equipment,  did  not  occur  on  time.  In  commenting 
on  the  AEHF  report,  DOD  noted  the  decision  to  accelerate  the  program  was  based 
on  a  satellite  constellation  gap  caused  by  the  loss  of  a  Milstar  satellite.  DOD  also 
stated  many  in  DOD  expressed  concern  about  the  risks,  but  believed  the  risk  was 
acceptable  based  on  information  known  at  the  time. 

2003  GAO  reported  that  at  the  time  DOD  decided  to  accelerate  the  program,  it  did 
not  have  the  funding  needed  to  support  the  activities  and  manpower  needed  to 
design  and  build  the  satellites  quicker.  The  lack  of  funding  also  contributed  to 
schedule  delays,  which  in  turn,  caused  more  cost  increases. 

2003  GAO  reported  that  the  program  demonstrated  most  technology  knowledge  at 
development  with  11  of  12  critical  technologies  having  reached  maturity 
according  to  best  practice  standards.  However,  the  program  office  did  not  project 
achieving  maturity  on  the  remaining  technology — the  phased  array  antenna —  by 
the  design  review  in  June  2004  and  did  not  have  a  backup  capability.  Program 
officials  assessed  the  software  development  for  the  mission  control  system  as 
moderate  risk  and  have  developed  a  risk  mitigation  strategy.  However,  until  these 
mitigation  actions  are  completed,  software  may  be  at  risk  for  unplanned  cost  and 
schedule  growth. 

2003  GAO  reported  that  significant  design  changes  affected  cost  and  delayed  the 
AEHF  schedule.  For  example,  software  growth  occurred  as  more  requirements 
were  added  and  as  the  design  of  the  system  stabilized.  These  increases  in  software 
requirements  for  both  the  satellite  and  the  mission  control  segments  increased  the 
software  cost  estimate  by  over  77  percent  or  about  $223  million. 
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The  GAO  provided  a  brief  synopsis  of  the  AEHF  program1  s  status  just  prior  to  the 

launch  of  AEHF-1  in  August  of  2010  (Government  Accountability  Office,  2010): 

According  to  the  program  office,  all  14  AEHF  critical  technologies  are  mature, 
with  all  either  flight-  qualified  through  test  and  demonstration  or  flight-  proven 
through  successful  mission  operations.  System-level  environmental  testing  for  the 
first  satellite  was  completed  in  July  2009.  The  AEHF‘s  design  appears  stable  with 
all  of  its  expected  design  drawings  released. 

. .  .during  initial  system  level  environmental  testing  for  the  first  and  second 
satellites,  several  flight  boxes  experienced  failures  due  to  defective  components 
that  required  removal,  repair,  and  reinstallation.  Because  of  the  number  of 
components  that  had  to  be  removed  and  reinstalled,  the  first  satellite  had  to 
undergo  an  additional  round  of  system-level  environmental  tests.  These  actions 
delayed  the  first  launch  almost  2  years  and  increased  program  cost.  According  to 
the  program  office,  the  additional  testing  was  successfully  completed  in  July 
2009.  The  second  satellite  also  completed  system  level  environmental  testing  in 
2009,  and  no  new  problems  or  issues  were  discovered. 

An  independent  review  team,  led  by  Mr.  Tom  Young,  assessed  the  NPOESS 

program  in  2009  (NPOESS  Independent  Review  Team,  2009): 

The  priorities  of  NOAA,  NASA  and  DOD/USAF  are  not  aligned:  The  DOD  has 
stated  that  while  the  program  should  continue  to  pursue  the  current  NPOESS 
requirements,  the  DOD  is  willing  to  accept  legacy  performance  (DMSP  and 
POES)  to  maintain  continuity,  cost  and  schedule  goals  and  is  not  willing  to 
provide  additional  funding  to  pursue  requirements  beyond  legacy.  NOAA  states 
that  legacy  perfonnance  would  be  a  step  back  in  today4  s  performance  because  of 
their  current  operational  use  of  NASA  research  satellites  that  are  well  beyond 
their  design  life 

NPOESS  is  being  managed  with  cost  as  the  most  important  parameter:  One 
observation  of  this  cost  priority  is  reflected  in  the  award  fee  structure  and  its 
emphasis  on  cost  control.  Successful  space  acquisition  requires  mission  success 
to  be  the  top  priority  not  cost  as  the  overarching  factor 

The  PEO  and  IPO  do  not  have  sufficient  space  systems  acquisition  expertise 
and  processes:  The  NPOESS  program  is  not  part  of  a  supporting  space  systems 
acquisition  center,  such  as  the  AF  Space  and  Missile  Systems  Center  (SMC)  or 
the  NASA  Goddard  Space  Flight  Center  (GSFC).  These  types  of  established 
space  acquisition  organizations  can  provide  institutional  knowledge,  robust 
infrastructure  support,  and  a  cadre  of  seasoned  space  systems  acquisition  experts 


75 


Funding  shortfalls  are  causing  the  IPO  to  make  short-sighted  decisions  to  cover 
VIIRS  cost  growth  and  stay  within  allocated  budget  at  a  significant  increase  to 
outyear  costs  and  program  risks:  While  the  IPO  has  no  choice  but  to  make  these 
decisions,  risk  is  being  deliberately  built  into  the  program  to  stay  within  allocated 
budget. 

The  current  budget  is  inadequate:  Budgeting  to  a  50-50  cost  estimate  leads  to 
insufficient  funding.  It  lacks  sufficient  management  reserve,  and  as  noted  in 
Finding  #6,  this  leads  to  programs  using  risk  as  its  management  reserve.  The 
current  budget  is  not  at  the  50/50  level.  The  most  probable  cost  is  at  the  80/20 
level  including  reserves 

The  SBIRS  program  has  been  reviewed  numerous  times  by  the  GAO  and  other 
entities.  The  2003  GAO  report  lists  the  following  findings  (Government  Accountability 
Office,  2003): 

History  of  moving  forward  without  sufficient  knowledge  to  ensure  that  the 
product  design  is  stable  and  meets  performance  requirements  and  that  adequate 
resources  are  available 

The  program  passed  its  critical  design  review  with  only  50  percent  of  its  design 
drawings  completed,  compared  to  90  percent  as  recommended  by  best  practices. 
Consequently,  several  design  modifications  were  necessary,  including  39  to  the 
first  of  two  infrared  sensors  to  reduce  excessive  noise  created  by  electromagnetic 
interference — a  threat  to  the  host  satellite^  functionality — delaying  delivery  of 
the  sensor  by  10  months  or  more 

The  program  was  too  immature  to  enter  the  system  design  and  development 
phase.  Program  activation  was  based  on  faulty  and  overly  optimistic  assumptions 
about  software  reuse  and  productivity  levels,  the  benefits  of  commercial  practices, 
management  stability,  and  the  level  of  understanding  of  requirements. 

The  complexity  of  developing  engineering  solutions  to  meet  system  requirements 
was  not  well  understood  by  program  and  contracting  officials.  The  systems 
integration  effort  was  significantly  underestimated  in  terms  of  complexity  and  the 
associated  impacts.  In  addition,  the  requirements  refinement  process  was  ad  hoc, 
creating  uncertainty  on  the  status  of  program  priorities  and  affecting  cost  and 
schedule. 

Breakdown  in  execution  and  management.  Overly  optimistic  assumptions  and 
unclear  requirements  eventually  overwhelmed  government  and  contractor 
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management.  The  2-year  delay  of  the  GEO  satellite  launches,  which  occurred  in 
1998,  contributed  to  management  instability  and  was  a  factor  in  the  Program 
Office  and  the  contractor  having  to  spend  25  of  the  first  60  months  of  the  contract 
on  replanning  activities. 

The  Department  of  Defense  was  tasked  by  Congress  to  provide  a  report  on  the 
SBIRS  High  progranTs  status  in  2005  (Office  of  the  Secretary  of  Defense,  2005)  which 
listed  the  following  findings: 

Latent  defects,  resulting  from  insufficient  product  assurance  activity  in  earlier 
design  and  production  activities. .  .lack  of  sound  system  engineering  processes  and 
procedures 

Insufficient  schedule  and  budget  to  ensure  robust  GEO  first  article  integration/ 
test... insufficient  time  scheduled  for  GEO  system  integration  and  test;  SPO 
concluded  the  ground  software  productivity  levels  were  optimistic;  the  flight 
software  architecture  was  not  sufficiently  defined  to  allow  software  coding;  and 
inadequate  on-orbit  checkout  time  was  planned.  Finally,  the  resources  and  tools 
for  simulations,  analysis,  and  troubleshooting  were  inadequate  and  required  more 
effort 

Process  escapes  due  to  human  error/insufficient  training/fragile 

processes. .  .improper  or  inadequate  processes,  insufficient  training,  questionable 

inspection  practices,  and/or  human  error  as  causal  factors.  Recent  events  include 

excess  debris  or  contamination  in  delivered  hardware,  improper  use  of  soldering 

materials,  improper  installation  of  thermal  blankets,  and  missing  test  procedure 

documentation 

A  poor  design  and  build  implementation  to  comply  with  the  EMI  specifications  of 
the  HEO  P/L. . .flawed  design  approach 

Faulty  hardware  and  software  design  of  the  HEO/GEO  flight  computers,  i.e.,  the 
single  board  computer  =balt‘  anomalies. .  .hardware  design  problem  with  a  control 
signal  on  an  Application-Specific  Integrated  Circuit  (ASIC) 

An  inadequate  architecture  design  and  a  flawed  flight  software  development  plan 
for  the  GEO  satellite^  Signal  Processing  Assembly  (SPA). .  .state  of  the  software 
architecture,  a  very  aggressive  contractor  schedule,  and  inadequate  planning 
The  GAO  again  assessed  the  SBIRS  High  program  in  2008  following  a  setback 

with  the  progranTs  flight  software  (Government  Accountability  Office,  2008): 
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While  DOD  has  estimated  that  the  SBIRS  program  will  be  delayed  by  1 5  months 
and  cost  $414  million  to  resolve  the  software  problems,  those  estimates  appear 
too  optimistic,  given  the  cost  and  schedule  risks  involved.  For  example,  SBIRS 
contractors1  report  low  confidence  that  software  can  be  produced  in  time  to  meet 
the  December  2009  satellite  launch  goal.  Further,  DOD  and  the  contractor  face 
significant  challenges  and  risks  that  could  result  in  more  time  and  money  being 
required  to  meet  program  goals,  to  include  the  bypassing  of  some  disciplined 
software  practices  that  add  risk  to  cost  and  schedule.  Finally,  as  of  August  2008, 
DOD  reported  that  SBIRS  was  already  behind  schedule  on  some  software 
development  efforts,  and  thousands  of  activities  remain  that  must  be  integrated 
and  tested  across  various  systems,  with  cost  and  schedule  implications,  if 
problems  or  unintended  consequences  occur. 

The  GAO‘s  annual  assessment  of  selected  programs  revealed  a  higher  level  of 

technical  maturity  for  SBIRS  High,  but  still  maintained  concerns  about  the  ability  of  the 

program  to  adhere  to  its  cost  and  schedule  (Government  Accountability  Office,  2010): 

The  SBIRS  High  program  began  system  development  in  1996  with  none  of  its 
three  critical  technologies  mature.  All  three  critical  technologies — the  infrared 
sensor,  thermal  management,  and  on-board  processing — are  now  mature  and  have 
been  demonstrated  in  at  least  a  relevant  environment.  Furthermore,  according  to 
the  program  office,  the  HEO  sensor‘s  on-orbit  performance  instills  confidence 
that  the  GEO  infrared  scanning  sensor  will  work  as  intended. 

According  to  program  officials,  99  percent  of  the  SBIRS  High  expected  design 
drawings  are  now  releasable.  However,  the  program  continues  to  experience 
design-related  problems,  and  more  could  emerge.  For  example,  flight  software 
design  problems  have  plagued  the  program  for  several  years,  causing  cost 
increases  and  schedule  delays,  and  the  program  may  still  be  underestimating  the 
amount  of  work  that  remains  to  resolve  the  issues. 

According  to  the  Defense  Contract  Management  Agency  (DCMA),  unplanned 
work  continues  to  be  a  challenge  for  the  software  development  effort  and  its  cost 
and  schedule  have  been  assessed  as  high  risk. 

The  SBIRS  High  program  remains  at  high  risk  for  cost  and  schedule  growth. 
DCMA  is  currently  projecting  over  $245  million  in  cost  overrun  from  the  current 
baseline  at  contract  completion.  This  amount  has  more  than  doubled  in  the  past 
year  and  continues  to  steadily  grow. 

Additional  contractor  cost  increases  and  schedule  delays  are  expected  due  in  part 
to  hardware  rework  on  the  first  satellite,  continued  difficulty  with  the  flight 
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software  development,  and  delays  in  integration  and  test  activities.  The  progranTs 
management  reserve —  funds  set  aside  to  address  unanticipated  problems —  will 
likely  be  depleted  before  the  first  GEO  satellite  launches,  and  additional  funding 
could  be  required  if  future  problems  occur.  Additional  schedule  delays  could  also 
occur  since  meeting  current  launch  estimates  depends  on  the  results  of  system- 
level  integration  tests. 

According  to  the  program  office,  the  first  GEO  integrated  payload  and  spacecraft 
successfully  completed  thermal  vacuum  (TVAC)  testing  in  November  2009. 
Program  officials  say  these  testing  results  give  them  high  confidence  that  the 
GEO  satellite  will  perform  similarly  to  the  successful  HEO  sensors,  noting  that 
HEO  TVAC  test  perfonnance  differed  only  slightly  from  its  on-orbit 
perfonnance. 

Program  officials  say  that  although  technical  issues  discovered  during  testing 
have  increased  program  cost,  parallel  activities  have  actually  minimized  program 
cost  and  schedule  growth.  They  further  stressed  that  mission  assurance  remains 
their  top  priority. 
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Appendix  B  -  Space  Acquisition  Cost  and  Schedule  Challenges 


Figure  1:  Differences  in  Total  Program  Costs  from  Program  Start  and  Most  Recent 
Estimates  (Fiscal  Year  2009) 
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Source:  GAO  analysis  of  DOO  data 


Legend:  SBIRS  =  Space  Based  Infrared  System;  GPS  =  Global  Positioning  System;  WGS  = 
Wideband  Global  SATCOM;  AEHF  =  Advanced  Extremely  High  Frequency;  NPOESS  =  National 
Polar-orbiting  Operational  Environmental  Satellite  System;  MUOS  =  Mobile  User  Objective  System 


Figure  8.  Space  Program  Cost  Growth  (Government  Accountability  Office,  2010) 
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Figure  2:  Differences  in  Unit  Costs  from  Program  Start  to  Most  Recent  Estimates 
(Fiscal  Year  2009) 
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Legend:  SBIRS  =  Space  Based  Infrared  System;  GPS  =  Global  Positioning  System;  WGS  = 
Wideband  Global  SATCOM;  AEHF  =  Advanced  Extremely  High  Frequency;  NPOESS  =  National 
Polar-orbiting  Operational  Environmental  Satellite  System;  MUOS  =  Mobile  User  Objective  System 


Figure  9.  Space  Program  Unit  Cost  Growth  (Government  Accountability  Office, 


2010) 
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Figure  3:  Differences  in  Total  Number  of  Months  to  IOC  from  Program  Start  and 
Most  Recent  Estimates 
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Legend:  SBIRS  =  Space  Based  Infrared  System:  GPS  =  Global  Positioning  System;  WGS  = 
Wideband  Global  SATCOM;  AEHF  =  Advanced  Extremely  High  Frequency;  NPOESS  =  National 
Polar-orbiting  Operational  Environmental  Satellite  System;  MUOS  =  Mobile  User  Objective  System. 


Figure  10.  Space  Program  Schedule  Delays  (Government  Accountability  Office, 

2010) 


82 


Appendix  C  -  A  Summary  of  Space  Acquisition  Findings  and  Root  Causes 


Table  35.  Space  Acquisition  Lessons  Learned 


System  or 
Subject 

Failure  //  Root  Cause 

Year 

Source 

AEHF 

early  phases  of  the  AEHF  program,  DOD 
substantially  and  frequently  altered  requirements; 

DOD  decided  to  accelerate  its  plans  to  build  the 

AEHF  satellites,  high  risk  schedule  that  turned  out  to 
be  overly  optimistic  and  highly  compressed-leaving 
little  room  for  error;  did  not  have  the  funding  needed 
to  support  the  activities  and  the  manpower  needed  to 
design  and  build  the  satellites  quicker 

2003 

GAO  -  Military  Space 
Operations:  Common 
Problems  and  Their 
Effects  on  Satellite  and 
Related  Acquisitions 

Distributed 

Common 

Ground- 

Surface 

System 

(DCGS) 

DOD  has  been  slow  to  plan  for  this  initiative  and  it 
has  not  addressed  important  questions  such  as  how 
and  when  systems  will  be  pared  down  and  modified 
as  well  as  how  the  initiative  will  be  funded. 

Moreover,  DOD  is  fielding  new  systems  and  new 
versions  of  old  systems  without  following  its  own 
certification  process 

2003 

GAO  -  Steps  Needed 
to  Ensure 

Interoperability  of 
Systems  That  Process 
Intelligence  Data 

Distributed 

Common 

Ground- 

Surface 

System 

(DCGS) 

incompatible  data  formats,  process  for  testing  and 
certifying  that  systems  will  be  interoperable  is  not 
working  effectively  //  lack  of  an  overarching  test 
plan 

2003 

GAO  -  Steps  Needed 
to  Ensure 

Interoperability  of 
Systems  That  Process 
Intelligence  Data 

GOES, 

lessons 

learned 

incorporation 

establish  realistic  cost  and  schedule  estimates 

2006 

GAO  -  Steps  Remain 
in  Incorporating 

Lessons  Learned  from 
Other  Satellite 

Programs 

GOES, 

lessons 

learned 

incorporation 

ensure  sufficient  technical  readiness  of  the  system's 
components  prior  to  key  decisions  //  GOES  I-M 
series,  NOAA  and  NASA  did  not  require 
engineering  analyses  prior  to  awarding  the 
development  contracts  in  order  to 
accelerate  the  schedule  and  launch  the  first  satellite. 
The  lack  of  these  studies  resulted  in  unexpected 
technical  issues  in  later  acquisition  phases — 
including  the  inability  of  the  original  instrument 
designs  to  withstand  the  temperature  variations  in  the 
geostationary  orbit 

2006 

GAO  -  Steps  Remain 
in  Incorporating 

Lessons  Learned  from 
Other  Satellite 

Programs 
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GOES, 

lessons 

learned 

incorporation 

provide  sufficient  management  at  government  and 
contractor  levels  //  The  key  drivers  of  poor 
management  included  inadequate  systems 
engineering  and  earned  value  management 
capabilities,  unsuitable  allocation  of  contract  award 
fees,  inadequate  levels  of  management  reserve,  and 
inefficient  decision-making  and  reporting  structure 
within  the  program  office 

2006 

GAO  -  Steps  Remain 
in  Incorporating 

Lessons  Learned  from 
Other  Satellite 

Programs 

GOES, 

lessons 

learned 

incorporation 

perform  adequate  senior  executive  oversight  to 
ensure  mission  success  //  lack  of  timely  decisions 
and  regular  involvement  of  senior  executive 
management  was  a  critical  factor  in  the  program‘s 
rapid  cost  and  schedule  growth 

2006 

GAO  -  Steps  Remain 
in  Incorporating 

Lessons  Learned  from 
Other  Satellite 

Programs 

GOES-R  risks 

hardware  that  is  to  be  used  for  the  ground  segment  is 
mature,  key  components  have  not  previously  been 
integrated.  Consequently,  if  the  components  do  not 
work  together,  the  program  might  have  to  procure 
separate  antennas,  which  would  impact  the 
program's  cost  and  schedule 

2009 

GAO  -  Acquisition  Is 
Under  Way,  but 
Improvements  Needed 
in  Management  and 
Oversight 

GOES-R  risks 

Advanced  Baseline  Imager  estimates  that  the 
instrument  is  over  50  percent  complete  and  reports 
that  it  has  experienced  technical  issues,  including 
problems  with  the  quality  of  components  in  the  focal 
plane  module,  mirrors,  and  telescope,  none  has  yet 
been  demonstrated  in  a  lab  or  test  environment,  the 
risk  remains  that  the  technologies  are  not  sufficiently 
mature 

2009 

GAO  -  Acquisition  Is 
Under  Way,  but 
Improvements  Needed 
in  Management  and 
Oversight 

Incentives  & 
Pressures 

lack  of  an  overall  investment  strategy;  tendency  to 
set  start  dates  for  programs  before  a  sound  business 
case  for  them  has  been  established;  DOD  starts  more 
programs  than  it  can  afford  and  rarely  prioritizes 
them  for  funding  purposes;  Such  an  approach  has 
cascading  effects — from  creating  negative  behaviors 
associated  with  competing  for  funds,  to  increasing 
technology  challenges,  to  creating  unanticipated  and 
disruptive  funding  shifts,  to  stretching  out  schedules 
in  order  to  accommodate  the  whole  portfolio  of 
space  programs 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  DOD  starts  more  programs  than  it  can  afford  over 
the  long  run,  forcing  programs  to  underestimate  costs 
and  overpromise  capability.  This  was  attributed  to 
both  the  Office  of  the  Secretary  of  Defense  and  the 

Air  Force.  The  September  11,  2001,  terror  attacks  on 
the  United  States  spurred  DOD  to  attempt  to  pursue 
even  more  satellite  programs,  believing  that  there 
was  now  a  greater  need  for  persistent  surveillance 
and  more  robust  communication  and  networking 
capabilities. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 
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Incentives  & 
Pressures 

•  When  faced  with  a  lower  budget,  senior  executives 
within  Office  of  the  Secretary  of  Defense  and  the  Air 
Force  would  rather  make  across-the  board  cuts  to  all 
space  programs  than  hard  decisions  as  to  which  ones 
to  keep  and  which  ones  to  cancel  or  cut  back. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Because  programs  are  funded  annually  and 
priorities  have  not  been  established,  competition  for 
funding  continues  over  time,  forcing  programs  to 
view  success  as  the  ability  to  secure  the  next 
installment  rather  than  the  end  goal  of  delivering 
capabilities  when  and  as  promised. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  More  often  than  not,  DOD  seeks  substantial  leaps 
in  capability  versus  incremental  leaps.  While  this 
approach  helps  a  program  to  gain  support,  it 
substantially  increases  the  technical  challenge  and 
the  level  of  unknowns  about  a  program  at  the  time  it 
is  started. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Having  to  continually  -sell”  a  program  also  creates 
incentives  to  suppress  bad  news  about  the  program's 
status  and  avoid  activities  that  uncover  bad  news. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Launching  demonstrators  in  space  is  a  good  way  to 
reduce  risks  and  learn  about  technologies  before 
starting  a  new  acquisition  program.  But  because  of 
the  high  cost  of  testing  technologies  in  space  and  the 
overall  competition  for  funding,  programs  are 
incentivized  not  to  pursue  this  approach.  At  the  same 
time,  resources  outside  acquisition  programs  devoted 
to  testing  in  an  operational  environment  are 
declining. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  DOD  faces  resource  shortages  beyond  funding 
because  it  starts  more  programs  than  it  can  afford. 
Principally,  it  does  not  have  a  sufficient  workforce  to 
support  space  acquisitions  or  experienced  program 
managers  to  guide  them. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

diverse  array  of  officials  and  organizations  involved 
with  the  acquisition  process,  tensions  between  the 

S&T  and  acquisition  communities  as  to  who  is  better 
suited  to  translate  technology  concepts  into  reality 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 
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Incentives  & 
Pressures 

•  The  lengthy  development  period  required  for  space 
systems  puts  pressure  on  program  managers  to 
continually  develop  technologies.  There  is  a  fear  that 
if  these  technologies  do  not  reach  maturity  during 
this  time  frame,  they  will  be  outdated  by  the  time  the 
satellites  are  ready  to  be  launched. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Once  a  program  has  formally  begun,  it  is  easier  to 
secure  current  and  future  years1  funding. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Satellites  tend  to  last  longer  than  expected,  and  they 
cannot  be  retrieved  for  upgrades,  putting  more 
pressure  on  programs  to  push  for  attaining  as  much 
technological  capability  as  possible  within  the 
acquisition  program. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  The  acquisition  community  does  not  believe  that 
labs  in  charge  of  developing  space  technologies 
adequately  understand  its  needs — in  terms  of 
capabilities  and  time  frames — and  would  rather 
pursue  its  own  goals. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Program  managers  also  believe  that  they  would 
have  more  control  over  technology  development  if  it 
was  conducted  by  contractors  who  answered  to  them 
rather  than  to  DOD  labs. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  DOD  has  not  had  an  effective  strategy  for  steering 
activities  within  the  S&T  community  to  ensure  that 
they  will  eventually  fit  in  with  acquisition  needs. 

(Note:  DOD  has  recently  developed  a  space  S&T 
strategy.  We  reported  on  this  effort  in  January  2005.) 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

pressures  resulting  from  short  tenures  among  staff 
critical  to  achieving  acquisition  success,  and 
difficulties  in  overseeing  contractors 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 
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Incentives  & 
Pressures 

•  Nonincumbent  contractors  are  often  able  to  submit 
a  lower  price  than  the  incumbent  because  they  can  be 
optimistic  without  being  challenged  by  DOD.  These 
optimistic  estimates  enable  them  to  win  new 
contracts.  At  the  same  time,  however, 
nonincumbents  are  not  necessarily  the  best 
organizations  to  carry  out  the  development  program, 
particularly  because  they  do  not  have  the  technical 
and  management  experience  associated  with  the 
legacy  system  being  replaced. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Industry  has  been  consolidated  to  a  point  where 
there  may  be  only  one  company  that  can  develop  a 
needed  component  for  a  satellite  system.  This  has 
enabled  contractors  to  hold  some  programs  hostage. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Program  managers  are  often  not  equipped  to 
understand  what  is  behind  a  contractors  proposal, 
particularly  because  contractors  are  not  likely  to 
disclose  technical  risks  and  highlight  other  negative 
aspects. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Industry  puts  pressure  on  programs  to  have 
contractors  develop  critical  technologies  within  an 
acquisition  environment  versus  having  the  labs  do  it. 
When  labs  build  technologies,  the  government 
allows  the  contractors  that  work  on  the  system  that 
would  ultimately  use  the  technologies  to  scrap  them 
in  favor  of  employing  their  own  methods  and 
expertise. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Program  managers  are  not  always  experienced 
enough  to  stand  up  to  contractors  when  development 
is  being  mismanaged.  Program  managers  also  may 
not  understand  the  best  ways  to  incentivize 
contractors  and  gain  insight  into  their  performance. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

Incentives  & 
Pressures 

•  Contractors  are  facing  workforce  pressures  similar 
to  those  experienced  by  the  government,  that  is,  not 
enough  technical  expertise  to  develop  highly 
complex  space  systems.  (Our  recent  report  on  space 
S&T  echoed  this  concern  as  well,  pointing  out  that 
several  studies  have  found  that  both  industry  and  the 
U.S.  government  face  substantial  shortages  of 
scientists  and  engineers  and  that  recruitment  of  new 
personnel  is  difficult  because  the  space  industry  is 
one  of  many  sectors  competing  for  the  limited 
number  of  trained  scientists  and  engineers.) 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 
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Incentives  & 
Pressures 

•  Some  space  programs  are  facing  pressures  related 
to  funding  and  technology  development  because  of 
an  expectation  widely  held  in  the  1990s  that  the 
commercial  space  market  would  experience  a  boom. 

At  the  time,  DOD  decreased  funding  for  some 
capabilities,  principally  space  launch,  assuming  the 
market  could  pay  for  a  portion  of  research  and 
development  and  that  economies  of  scale  would 
result.  It  also  relied  on  the  commercial  sector  to 
develop  knowledge  about  production  of  satellites 
that  eventually  were  purchased  as  part  of  the 

Wideband  Gap  filler  Satellite  program.  However, 
when  anticipated  commercial  orders  using  the  same 
technologies  did  not  pan  out,  the  government 
experienced  unanticipated  schedule  delays. 

2005 

GAO  -  Defense 
Acquisitions: 

Incentives  and 

Pressures  That  Drive 
Problems  Affecting 
Satellite  and  Related 
Acquisitions 

NPOESS 

requirements  setting  problems  attributable  to  the 
broad  base  of  internal  customers  each  agency  has 
and  the  diversity  of  requirements  that  needed  to  be 
met 

2003 

GAO  -  Military  Space 
Operations:  Common 
Problems  and  Their 
Effects  on  Satellite  and 
Related  Acquisitions 

NPOESS 

NPOESS  is  being  managed  with  cost  as  the  most 
important  parameter:  One  observation  of  this  cost 
priority  is  reflected  in  the  award  fee  structure  and  its 
emphasis  on  cost  control.  Successful  space 
acquisition  requires  mission  success  to  be  the  top 
priority  not  cost  as  the  overarching  factor 

2009 

NPOESS  Independent 
Review  Team 

NPOESS 

The  EXCOM  process  is  ineffective:  The  EXCOM  is 
intended  to  be  a  decision  body  to  provide 
streamlined  direction  to  the  PEO.  The  current  DOD 
EXCOM  representative  has  not  been  delegated  the 
proper  authority  from  the  Defense  Acquisition 
Executive  (DAE),  who  is  also  the  NPOESS 

Milestone  Decision  Authority  (MDA),  and  decisions 
require  an  additional  meeting  and  coordination  to  be 
finalized 

2009 

NPOESS  Independent 
Review  Team 

NPOESS 

The  PEO  and  IPO  do  not  have  sufficient  space 
systems  acquisition  expertise  and  processes:  The 
NPOESS  program  is  not  part  of  a  supporting  space 
systems  acquisition  center,  such  as  the  AF  Space  and 
Missile  Systems  Center  (SMC)  or  the  NASA 

Goddard  Space  Flight  Center  (GSFC).  These  types 
of  established  space  acquisition  organizations  can 
provide  institutional  knowledge,  robust  infrastructure 
support,  and  a  cadre  of  seasoned  space  systems 
acquisition  experts 

2009 

NPOESS  Independent 
Review  Team 

NPOESS 

Funding  shortfalls  are  causing  the  IPO  to  make 
short-sighted  decisions  to  cover  VIIRS  cost  growth 
and  stay  within  allocated  budget  at  a  significant 
increase  to  outyear  costs  and  program  risks:  While 
the  IPO  has  no  choice  but  to  make  these  decisions, 
risk  is  being  deliberately  built  into  the  program  to 
stay  within  allocated  budget. 

2009 

NPOESS  Independent 
Review  Team 
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NPOESS 

The  priorities  of  NOAA,  NASA  and  DOD/USAF  are 
not  aligned:  The  DOD  has  stated  that  while  the 
program  should  continue  to  pursue  the  current 
NPOESS  requirements,  the  DOD  is  willing  to  accept 
legacy  performance  (DMSP  and  POES)  to  maintain 
continuity,  cost  and  schedule  goals  and  is  not  willing 
to  provide  additional  funding  to  pursue  requirements 
beyond  legacy.  NOAA  states  that  legacy 
performance  would  be  a  step  back  in  today's 
performance  because  of  their  current  operational  use 
of  NASA  research  satellites  that  are  well  beyond 
their  design  life 

2009 

NPOESS  Independent 
Review  Team 

NPOESS 

The  current  budget  is  inadequate:  Budgeting  to  a  50- 
50  cost  estimate  leads  to  insufficient  funding.  It  lacks 
sufficient  management  reserve,  and  as  noted  in 

Finding  #6,  this  leads  to  programs  using  risk  as  its 
management  reserve.  The  current  budget  is  not  at  the 
50/50  level.  The  most  probable  cost  is  at  the  80/20 
level  including  reserves 

2009 

NPOESS  Independent 
Review  Team 

NPOESS 

Committee  lacks  the  membership  and  leadership 
needed  to  effectively  and  efficiently  oversee  and 
direct  the  program.  Specifically,  the  DOD 

Committee  member  with  acquisition  authority  does 
not  attend  Executive  Committee  meetings — and 
sometimes  contradicts  the  Committee's  decisions, 
the  Committee  does  not  track  its  action  items  to 
closure,  and  many  of  the  Committee's  decisions  do 
not  achieve  desired  outcomes  //  DOD's  acquisition 
authority  has  never  attended  an  Executive 

Committee  meeting.  This  individual  delegated  the 
responsibility  for  attending  the  meetings — but  not 
the  authority  to  make  acquisition  decisions — to  the 
Under  Secretary  of  the  Air  Force  //  agreements 
between  committee  members  have  been  overturned 
by  the  acquisition  authority,  leading  to  significant 
delays  //  NPOESS  Executive  Committee  generally 
took  immediate  action  to  mitigate  the  risks  that  were 
brought  before  them;  however,  a  majority  of  these 
actions  were  not  effective — that  is,  they  did  not  fully 
resolve  the  underlying  issues  or  result  in  a  successful 
outcome  //  interagency  disagreements  and  differing 
priorities 

2009 

GAO  -  With  Costs 
Increasing  and  Data 
Continuity  at  Risk, 
Improvements  Needed 
in  Tri -agency  Decision 
Making 

NPOESS 

Specifically,  ongoing  challenges  with  VIIRS 
development,  design,  and  workmanship  have  led  to 
additional  cost  overruns  and  delayed  the  instrument's 
delivery  to  NPP 

2009 

GAO  -  With  Costs 
Increasing  and  Data 
Continuity  at  Risk, 
Improvements  Needed 
in  Tri -agency  Decision 
Making 
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NPOESS 

problems  discovered  during  environmental  testing  on 
CrIS  led  the  contractor  to  further  delay  its  delivery  to 
NPP  and  added  further  unanticipated  costs  to  the 
program 

2009 

GAO  -  With  Costs 
Increasing  and  Data 
Continuity  at  Risk, 
Improvements  Needed 
in  Tri-agency  Decision 
Making 

NPOESS  risks 

Progress  Has  Been  Made  in  Establishing  an  Effective 
NPOESS  Management  Structure,  but  Executive 
Turnover  Increases  Risks  and  Staffing  Problems 
Remain 

2007 

GAO- 

ENVIRONMENTAL 
SATELLITE 
ACQUISITIONS 
Progress  and 

Challenges 

NPOESS  risks 

Space  Segment — Progress  Made,  but  Key  Sensors 
Continue  to  Face  Major  Risks  //  VIIRS  -  completed 
environmental  tests  of  VIIRS ‘s  engineering  design 
unit  (a  prototype)  (1)  band-to-band  co-registration, 
an  issue  in  which  band  registration  shifts  with 
different  temperatures;  (2)  cross-talk,  which  involves 
information  from  sensor  cells  leaking  into  other 
cells;  and  (3)  line-spread  function  issues,  in  which 
the  instrument's  focus  changes  with  changes  in 
temperature,  CrIS  -  Development  of  CrIS  was  put  on 
hold  in  October  2006  when  the  flight  unit  designated 
to  go  on  NPP  experienced  a  major  structural  failure 
during  its  vibration  test 

2007 

GAO- 

ENVIRONMENTAL 
SATELLITE 
ACQUISITIONS 
Progress  and 

Challenges 

SBIRS 

program's  history  of  moving  forward  without 
sufficient  knowledge  to  ensure  that  the  product 
design  is  stable  and  meets  performance  requirements 
and  that  adequate  resources  are  available 

2003 

GAO  -  Despite 
Restructuring,  SBIRS 
High  Program 

Remains  at  Risk  of 

Cost  and  Schedule 
Overruns 

SBIRS 

program  passed  its  critical  design  review  with  only 

50  percent  of  its  design  drawings  completed, 
compared  to  90  percent  as  recommended  by  best 
practices.  Consequently,  several  design 
modifications  were  necessary,  including  39  to  the 
first  of  two  infrared  sensors  to  reduce  excessive 
noise  created  by  electromagnetic  interference — a 
threat  to  the  host  satellite's  functionality — delaying 
delivery  of  the  sensor  by  10  months  or  more 

2003 

GAO  -  Despite 
Restructuring,  SBIRS 
High  Program 

Remains  at  Risk  of 

Cost  and  Schedule 
Overruns 

SBIRS 

testing  of  the  first  infrared  sensor  revealed  several 
deficiencies  in  the  flight  software  involving  the 
sensor's  ability  to  maintain  earth  coverage  and  track 
missiles  while  orbiting  the  earth  (flight  software  still 
major  program  risk) 

2003 

GAO  -  Despite 
Restructuring,  SBIRS 
High  Program 

Remains  at  Risk  of 

Cost  and  Schedule 
Overruns 

SBIRS 

The  program  was  too  immature  to  enter  the  system 
design  and  development  phase.  Program  activation 
was  based  on  faulty  and  overly  optimistic 
assumptions  about  software  reuse  and  productivity 
levels,  the  benefits  of  commercial  practices, 
management  stability,  and  the  level  of  understanding 

2003 

GAO  -  Despite 
Restructuring,  SBIRS 
High  Program 

Remains  at  Risk  of 

Cost  and  Schedule 
Overruns 

90 


of  requirements. 

SBIRS 

The  complexity  of  developing  engineering  solutions 
to  meet  system  requirements  was  not  well 
understood  by  program  and  contracting  officials.  The 
systems  integration  effort  was  significantly 
underestimated  in  terms  of  complexity  and  the 
associated  impacts.  In  addition,  the  requirements 
refinement  process  was  ad  hoc,  creating  uncertainty 
on  the  status  of  program  priorities  and  affecting  cost 
and  schedule. 

2003 

GAO  -  Despite 
Restructuring,  SBIRS 
High  Program 

Remains  at  Risk  of 

Cost  and  Schedule 
Overruns 

SBIRS 

Breakdown  in  execution  and  management.  Overly 
optimistic  assumptions  and  unclear  requirements 
eventually  overwhelmed  government  and  contractor 
management.  The  2-year  delay  of  the  GEO  satellite 
launches,  which  occurred  in  1998,  contributed  to 
management  instability  and  was  a  factor  in  the 
Program  Office  and  the  contractor  having  to  spend 

25  of  the  first  60  months  of  the  contract  on 
replanning  activities. 

2003 

GAO  -  Despite 
Restructuring,  SBIRS 
High  Program 

Remains  at  Risk  of 

Cost  and  Schedule 
Overruns 

SBIRS 

latent  defects,  resulting  from  insufficient  product 
assurance  activity  in  earlier  design  and  production 
activities  //  lack  of  sound  system  engineering 
processes  and  procedures 

2005 

DOD  -  Status  of  the 
Space  Based  Infrared 
System  Program, 

Report  to  the  Defense 
and  Intelligence 
Committees 

SBIRS 

insufficient  schedule  and  budget  to  ensure  robust 

GEO  first  article  integration  /  test  //  insufficient  time 
scheduled  for  GEO  system  integration  and  test;  SPO 
concluded  the  ground  software  productivity  levels 
were  optimistic;  the  flight  software  architecture  was 
not  sufficiently  defined  to  allow  software  coding; 
and  inadequate  on-orbit  checkout  time  was  planned. 
Finally,  the  resources  and  tools  for  simulations, 
analysis,  and  troubleshooting  were  inadequate  and 
required  more  effort 

2005 

DOD  -  Status  of  the 
Space  Based  Infrared 
System  Program, 

Report  to  the  Defense 
and  Intelligence 
Committees 

SBIRS 

process  escapes  due  to  human  error  /  insufficient 
training  /  fragile  processes  //  improper  or  inadequate 
processes,  insufficient  training,  questionable 
inspection  practices,  and/or  human  error  as  causal 
factors.  Recent  events  include  excess  debris  or 
contamination  in  delivered  hardware,  improper  use 
of  soldering  materials,  improper  installation  of 
thermal  blankets,  and  missing  test  procedure 
documentation 

2005 

DOD  -  Status  of  the 
Space  Based  Infrared 
System  Program, 

Report  to  the  Defense 
and  Intelligence 
Committees 
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SBIRS 

A  poor  design  and  build  implementation  to  comply 
with  the  EMI  specifications  of  the  HEO  P/L  //  flawed 
design  approach 

2005 

DOD  -  Status  of  the 
Space  Based  Infrared 
System  Program, 

Report  to  the  Defense 
and  Intelligence 
Committees 

SBIRS 

Faulty  hardware  and  software  design  of  the 

HEO/GEO  flight  computers,  i.e.,  the  single  board 
computer  _halt‘  anomalies  //  hardware  design 
problem  with  a  control  signal  on  an  Application- 
Specific  Integrated  Circuit  (ASIC) 

2005 

DOD  -  Status  of  the 
Space  Based  Infrared 
System  Program, 

Report  to  the  Defense 
and  Intelligence 
Committees 

SBIRS 

An  inadequate  architecture  design  and  a  flawed  flight 
software  development  plan  for  the  GEO  satellite^ 
Signal  Processing  Assembly  (SPA)  //  state  of  the 
software  architecture,  a  very  aggressive  contractor 
schedule,  and  inadequate  planning 

2005 

DOD  -  Status  of  the 
Space  Based  Infrared 
System  Program, 

Report  to  the  Defense 
and  Intelligence 
Committees 

SBIRS 

flight  software  for  the  first  satellite  underwent  testing 
and  failed;  timing  of  stored  programs  //  test  beds  that 
had  matured  in  parallel  with  the  flight  software  and 
hardware,  making  it  difficult  to  distinguish  between 
test  bed  and  software  issues;  oversubscription  of  test 
beds  and  lack  of  simulation  resources  that  precluded 
them  from  checking  out  high-risk  areas  (timing,  and 
stored  programs);  insufficient  modeling  of  timing, 
and  analysis  of  stored  program  implementation, 
which  might  have  shed  light  earlier  on  lack  of 
robustness 

2008 

GAO  -  DOD‘s  Goals 
for 

Resolving  Space 

Based  Infrared  System 
Software  Problems 

Are  Ambitious 

SBIRS 

flight  software  for  the  first  satellite  underwent  testing 
and  failed;  distribution  of  control  between  processors 
//  test  beds  that  had  matured  in  parallel  with  the 
flight  software  and  hardware,  making  it  difficult  to 
distinguish  between  test  bed  and  software  issues; 
oversubscription  of  test  beds  and  lack  of  simulation 
resources  that  precluded  them  from  checking  out 
high-risk  areas  (timing,  and  stored  programs); 
insufficient  modeling  of  timing,  and  analysis  of 
stored  program  implementation,  which  might  have 
shed  light  earlier  on  lack  of  robustness 

2008 

GAO  -  DOD‘s  Goals 
for 

Resolving  Space 

Based  Infrared  System 
Software  Problems 

Are  Ambitious 

SBIRS 

flight  software  for  the  first  satellite  underwent  testing 
and  failed;  failure  at  the  hardware  interface  level  //  // 
test  beds  that  had  matured  in  parallel  with  the  flight 
software  and  hardware,  making  it  difficult  to 
distinguish  between  test  bed  and  software  issues; 
oversubscription  of  test  beds  and  lack  of  simulation 
resources  that  precluded  them  from  checking  out 
high-risk  areas  (timing,  and  stored  programs); 
insufficient  modeling  of  timing,  and  analysis  of 
stored  program  implementation,  which  might  have 
shed  light  earlier  on  lack  of  robustness 

2008 

GAO  -  DOD‘s  Goals 
for 

Resolving  Space 

Based  Infrared  System 
Software  Problems 

Are  Ambitious 
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SBIRS 

weaknesses  in  management  responsibility, 
accountability  and  organizational  structure;  Air 

Force  has  limited  management  control  over  SBIRS 

2008 

GAO  -  DOD‘s  Goals 
for 

Resolving  Space 

Based  Infrared  System 
Software  Problems 

Are  Ambitious 

SBIRS 

Lockheed  Martin ‘s  program  process  discipline  is 
poor 

2008 

GAO  -  DOD‘s  Goals 
for 

Resolving  Space 

Based  Infrared  System 
Software  Problems 

Are  Ambitious 

SBIRS 

Adversarial  relationships  exist  between  Air  Force 
and  Lockheed  Martin 

2008 

GAO  -  DOD‘s  Goals 
for 

Resolving  Space 

Based  Infrared  System 
Software  Problems 

Are  Ambitious 

SBIRS 

Government  organizational  structure  is  flawed 
because  cost  and  schedule  responsibilities  are 
separated 

2008 

GAO  -  DOD‘s  Goals 
for 

Resolving  Space 

Based  Infrared  System 
Software  Problems 

Are  Ambitious 

SBIRS 

Focal  point  for  FSS  completion  is  needed 

2008 

GAO  -  DOD‘s  Goals 
for 

Resolving  Space 

Based  Infrared  System 
Software  Problems 

Are  Ambitious 

SBIRS 

reasons  for  the  delay  include  poor  government 
oversight  of  the  contractor,  technical  complexities, 
and  rework.  The  program  continues  to  struggle  with 
flight  software  development,  and  during  testing  last 
year,  officials  discovered  hardware  defects  on  the 
first  GEO  satellite,  though  the  program  reports  that 
they  have  been  resolved 

2010 

GAO  -  DOD  Poised  to 
Enhance  Space 
Capabilities,  but 
Persistent  Challenges 
Remain  in  Developing 
Space  Systems 

SBIRS-High 

•  Cost-driven, 

•  Underfunded, 

•  Optimistic  contractor  proposal, 

•  Uncontrolled  requirements, 

•  Limited  program  manager  authority  and  capability, 

•  Funding  instability  (four  replans), 

•  Program  manager  instability  (four  government  and 
four  industry  program 

managers),  and 

•  Failure  to  implement  -best  practices.” 

2003 

Defense  Science  Board 
-  Acquisition  of 

National  Security 

Space  Programs 
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SBIRS-High 

independent  review  team  chartered  by  DOD  to 
examine  the  reasons  behind  cost  and  scheduling 
problems  in  the  SBIRS-High  program  reported  that  a 
key  root  cause  was  that  system  requirements  were 
not  well-understood  when  the  program  began  and  as 
it  evolved;  requirements  setting  process  was  often 
adhoc  with  many  decisions  being  deferred  to  the 
contractor.  The  review  team  also  found  that  the 
program  was  too  immature  to  enter  system  design 
and  development.  Further,  there  was  too  much 
instability  on  the  program  after  the  contract  award 

2003 

GAO  -  Military  Space 
Operations:  Common 
Problems  and  Their 
Effects  on  Satellite  and 
Related  Acquisitions 

Space  Radar 

•  Knowledge  point  1 :  A  match  must  be  made 
between  the  customer's  requirements  and  the 
developer's  available  resources  before  product 
development  starts.  As  noted  earlier,  DOD  plans  to 
start  SBR  product  development  in  2006. 

•  Knowledge  point  2:  The  product's  design  must  be 
stable  and  must  meet  performance  requirements 
before  initial  manufacturing  begins. 

•  Knowledge  point  3 :  The  product  must  be 
producible  within  cost,  schedule,  and  quality  targets 
and  demonstrated  to  be  reliable  before  production 
begins. 

2004 

GAO  -  DEFENSE 
ACQUISITIONS 
Space-Based  Radar 
Effort  Needs 

Additional  Knowledge 
before  Starting 
Development 

Space  Radar 

A  defined  requirements  approval  process  helps 
decision  makers  resolve  disagreements  that  may 
occur  and  ensure  they  will  remain  committed  to 
their  decisions  after  formal  approval.  Based  on  our 
past  reports  on  uncovering  problems  and  our  best 
practice  work,  we  believe  that  the  steps  in  a  formal 
approval  process  include: 

•  explaining  how  decision  makers'  requirements  and 
comments  are  obtained  and  addressed; 

•  identifying  the  officials  and/or  the  organizations 
responsible  for  taking  specific  approval  action; 

•  establishing  a  mechanism  and  time  frame  for 
providing  approval  or  disapproval; 

•  establishing  a  system  for  addressing  unresolved 
issues  as  they  relate  to  key  program  documentation; 
and 

•  assessing  changes  to  approved  requirements  based 
on  their  effect  on  the  program's  cost  and  schedule. 

2004 

GAO  -  DEFENSE 
ACQUISITIONS 
Space-Based  Radar 
Effort  Needs 

Additional  Knowledge 
before  Starting 
Development 

94 


Space  Radar 

it  is  expected  that  some  critical  SBR  technologies 
will  not  be  mature  when  product  development  starts, 
that  is,  not  tested  in  a  relevant  or  operational 
environment.  Typical  outcomes  of  this  lack  of 
knowledge  are  significant  cost  and  schedule 
increases  because  of  the  need  to  fix  problems  later  in 
development.  Furthermore,  TCA,  a  new,  more  robust 
communications  infrastructure  that  could  transmit 
SBR‘s  imagery  data  much  more  quickly  than  the 
current  infrastructure,  is  facing  uncertainties. 
Specifically,  one  of  TCA‘s  primary  components,  the 
Transformational  Satellite,  may  not  be  ready  in  time 
to  support  SBR.  However,  if  DOD  begins  product 
development  with  less  than  mature  technologies  and 
without  knowing  the  availability  of  TCA,  accurate 
cost  estimates  for  SBR  will  be  much  more  difficult  to 
prepare 

2004 

GAO  -  DEFENSE 
ACQUISITIONS 
Space-Based  Radar 
Effort  Needs 

Additional  Knowledge 
before  Starting 
Development 

Space, 

Common 

Problems 

requirements  definition  and  control  issues 

2003 

Defense  Science  Board 
-  Acquisition  of 

National  Security 

Space  Programs 

Space, 

Common 

Problems 

Cost  has  replaced  mission  success  as  the  primary 
driver  in  managing  acquisition  processes,  resulting  in 
excessive  technical  and  schedule  risk 

2003 

Defense  Science  Board 
-  Acquisition  of 

National  Security 

Space  Programs 

Space, 

Common 

Problems 

overall  underappreciation  of  the  importance  of 
appropriately  staffed  and  trained  system  engineering 
staffs  to  manage  the  technologically  demanding  and 
unique  aspects  of  space  programs;  Government 
capabilities  to  lead  and  manage  the  acquisition 
process  have  seriously  eroded 

2003 

Defense  Science  Board 
-  Acquisition  of 

National  Security 

Space  Programs 

Space, 

Common 

Problems 

The  space  acquisition  system  is  strongly  biased  to 
produce  unrealistically  low  cost  estimates  throughout 
the  acquisition  process.  These  estimates  lead  to 
unrealistic  budgets  and  unexecutable  programs; 
widespread  lack  of  budget  reserves  required  to 
implement  high  risk  programs  on  schedule; 
unhealthy  cost  bias  in  proposal  evaluation  // 
government  typically  has  invested  significantly  in 
capital  and  intellectual  resources  for  the  incumbent. 
When  the  incumbent  loses,  both  capital  resources 
and  the  mature  engineering  and  management 
capability  are  lost.  A  similar  investment  must  be 
made  in  the  new  contractor  team.  The  government 
pays  for  purchase  and  installation  of  specialized 
equipment,  as  well  as  fit-out  of  manufacturing  and 
assembly  spaces  that  are  tailored  to  meet  the  needs  of 
the  program 

2003 

Defense  Science  Board 
-  Acquisition  of 

National  Security 

Space  Programs 

Space, 

Common 

Problems 

While  the  space  industrial  base  is  adequate  to 
support  current  programs,  long-term  concerns  exist; 
Industry  has  failed  to  implement  proven  practices  on 
some  programs 

2003 

Defense  Science  Board 
-  Acquisition  of 

National  Security 

Space  Programs 
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Space, 

Common 

Problems 

requirements  for  what  the  satellite  needed  to  do  and 
how  well  it  must  perform  were  not  adequately 
defined  at  the  beginning  of  a  program  or  were 
changed  significantly  once  the  program  had  already 
begun.  This  made  it  more  difficult  for  programs  to 
ensure  that  they  could  match  their  requirements  to 
their  resources  (in  terms  of  money,  time,  and 
technology).  The  more  requirements  were  added  or 
changed,  the  more  that  cost  and  schedule  increased 

•  Program  did  not  adequately  define  requirements 

•  Unresolved  conflicts  among  users  on  requirements 

•  Frequent  changes  made  to  requirements  after 
product  development  began 

//  schedule-driven  versus  a  knowledge-driven 
approach;  diverse  array  of  organizations  with 
competing  interests  involved  in  overall  satellite 
development,  no  high-level  official  within  the  Office 
of  the  Secretary  of  Defense  dedicated  to  developing 
and  implementing  an  overall  investment  strategy  for 
space;  attempted  to  satisfy  all  requirements  in  a 
single  step,  regardless  of  the  design  challenge  or  the 
maturity  of  technologies  to  achieve  the  full  capability 

2003 

GAO  -  Military  Space 
Operations:  Common 
Problems  and  Their 
Effects  on  Satellite  and 
Related  Acquisitions 

Space, 

Common 

Problems 

investment  practices  were  weak.  At  times,  programs 
did  not  explore  potentially  more  cost-effective 
investment  approaches.  Once  they  settled  on  an 
approach,  programs  often  did  not  develop  realistic 
cost  estimates.  From  a  broader  perspective, 
investments  in  programs  were  not  made  in 
accordance  with  an  overall  space  investment  strategy 
for  DOD.  Funds  were  sometimes  shifted  from 
healthier  programs  to  pay  for  weaker  ones.  Further, 
according  to  DOD  officials,  decisions  external  to  the 
program  office  were  sometimes  imposed  that 
resulted  in  unexpected  funding  cuts 

•  Program  did  not  adequately  analyze  investment 
alternatives 

•  Cost  and/or  schedule  estimates  were  optimistic 

•  Funding  was  unstable 

//  schedule-driven  versus  a  knowledge-driven 
approach;  diverse  array  of  organizations  with 
competing  interests  involved  in  overall  satellite 
development,  no  high-level  official  within  the  Office 
of  the  Secretary  of  Defense  dedicated  to  developing 
and  implementing  an  overall  investment  strategy  for 
space 

2003 

GAO  -  Military  Space 
Operations:  Common 
Problems  and  Their 
Effects  on  Satellite  and 
Related  Acquisitions 
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Space, 

Common 

Problems 

acquisition  strategies  were  poorly  executed.  For 
example,  competition  was  reduced  for  the  sake  of 
schedule  or  DOD  did  not  adequately  oversee 
contractors.  At  times,  contract  type  was  not  suitable 
for  the  work  being  done 

•  Level  of  competition  was  reduced  or  eliminated 

•  Contract  type  was  not  suitable  for  work  being  done 

•  Poor  oversight  over  contractors 

//  schedule-driven  versus  a  knowledge-driven 
approach 

2003 

GAO  -  Military  Space 
Operations:  Common 
Problems  and  Their 
Effects  on  Satellite  and 
Related  Acquisitions 

Space, 

Common 

Problems 

programs  did  not  always  ensure  that  technologies 
were  mature  before  making  heavy  investments  in  the 
program.  This  often  caused  cost  and  schedule 
increases  due  to  the  need  to  fix  problems  later  in 
development.  A  continuing  problem  is  that  software 
needs  are  poorly  understood  at  the  beginning  of  a 
program 

•  Technology  not  sufficiently  mature  at  program  start 

•  Software  needs  poorly  understood 

•  Testing  compressed,  skipped,  or  done  concurrently 
with  production 

//  schedule-driven  versus  a  knowledge-driven 
approach;  attempted  to  satisfy  all  requirements  in  a 
single  step,  regardless  of  the  design  challenge  or  the 
maturity  of  technologies  to  achieve  the  full  capability 

2003 

GAO  -  Military  Space 
Operations:  Common 
Problems  and  Their 
Effects  on  Satellite  and 
Related  Acquisitions 

Space, 

Common 

Problems 

There  are  insufficient  numbers  of  technically 
competent  and  experienced  space  acquisition 
personnel  to  execute  the  responsibilities  of  the  Space 
and  Missile  Systems  Center  (SMC)  and  the  National 
Reconnaissance  Office  (NRO)  //  The  reduced 
availability  of  government  personnel  with  the 
necessary  technical  competence  has  sharply  reduced 
the  government1  s  capability  to  acquire  space  systems 
and  is  believed  by  many  experts  to  be  a  major  cause 
of  acquisition  program  failures 

2008 

Institute  for  Defense 
Analyses  -  Leadership, 
Management,  and 
Organization  for 
National  Security 

Space 

Space, 

Common 

Problems 

Lax  requirements  discipline,  technical  performance 
problems,  cost  growth,  and  schedule  delays  have 
plagued  U.S.  space  programs  //  existing  leadership 
and  management  practices  have  failed  to  define, 
fund,  and  execute  new  satellite  programs.  Strong 
management  is  needed  to  implement  proven 
acquisition  practices 

2008 

Institute  for  Defense 
Analyses  -  Leadership, 
Management,  and 
Organization  for 
National  Security 

Space 

97 


Space, 

Common 

Problems 

leadership  for  National  Security  Space  is  currently 
fragmented  and  unfocused.  Authorities  and 
responsibilities  are  spread  across  numerous 
organizations,  including  many  within  the  Office  of 
the  Secretary  of  Defense  (OSD)  [Under  Secretary  of 
Defense  (USD)/Intelligence;  USD/Acquisition, 
Technology,  and  Logistics;  USD/Policy;  and  the 
Assistant  Secretary  of  Defense  ( ASD)/Networks  & 
Information  Integration],  USAF,  USN,  USA,  USMC, 
DARPA,  MDA,  and  NRO.  Although  the  Secretary  of 
the  Air  Force  is  the  DOD  Executive  Agent  for  Space, 
its  authorities  have  been  diminished  from  those 
envisioned  by  the  2001  Space  Commission. 

Moreover,  as  perceived  by  many,  its  stewardship  of 
Space  does  not  enjoy  the  same  priority  as  other 
traditional  Air  Force  missions.  The  customers  who 
use  Space  capabilities  observe  that  there  is  no 
responsible  official  who  looks  across  all  the  available 
resources  and  capabilities  to  seek  the  best  solution, 
whether  from  the  military,  intelligence,  civilian,  or 
commercial  sector.  This  represents  a  critical  need 

2008 

Institute  for  Defense 
Analyses  -  Leadership, 
Management,  and 
Organization  for 
National  Security 

Space 

TSAT 

When  DOD  established  initial  goals  for  the  TSAT 
program,  it  lacked  sufficient  knowledge  about  key 
critical  technologies.  Our  past  work  has  shown  that  a 
knowledge-based  model  leads  to  better  acquisition 
outcomes.  This  model  can  be  broken  down  into  three 
cumulative  knowledge  points  for  technology 
maturity,  design  maturity,  and  production  maturity. 

At  the  first  knowledge  point,  a  match  is  made 
between  a  customer's  requirements  and  the  product 
developer's  available  resources  in  terms  of  technical 
knowledge,  time,  money,  and  capacity.  We  have  also 
reported  that  starting  a  complex  program  like  TSAT 
with  immature  technologies  can  lead  to  poor 
program  performance  and  outcomes. 

2006 

GAO  -  SPACE 
ACQUISITIONS 

DOD  Needs 

Additional  Knowledge 
as  it  Embarks  on  a 

New  Approach  for 

Transformational 

Satellite 

The  table  above  contains  text  from  the  following  sources:  (Government  Accountability  Office,  2010) 
(Government  Accountability  Office,  2003)  (Government  Accountability  Office,  2004)  (Government 
Accountability  Office,  2003)  (Government  Accountability  Office,  2005)  (Government  Accountability 
Office,  2007)  (Government  Accountability  Office,  2009)  (Government  Accountability  Office,  2006) 
(Government  Accountability  Office,  2003)  (Government  Accountability  Office,  2009)  (Government 
Accountability  Office,  2006)  (Government  Accountability  Office,  2008)  (Institute  for  Defense  Analyses, 
2008)  (NPOESS  Independent  Review  Team,  2009)  (Office  of  the  Undersecretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  2003) 
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Appendix  D  -  Recommendations  for  Space  Acquisition  Improvement 

The  2003  Defense  Science  Board  report  laid  out  a  number  of  steps  necessary  to 
correct  the  space  acquisition  deficiencies  previously  discussed.  The  combination  of 
space  acquisition  lessons  learned  and  recommendations  for  improvement  were  used  to 
compose  the  maturity  model.  The  report  cited  the  following  steps  (Office  of  the 
Undersecretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  2003): 

1 .  The  Under  Secretary  of  the  Air  Force/Director  National  Reconnaissance  Office 
(USecAF/DNRO)  should  establish  mission  success  as  the  guiding  principle  in  all 
space  systems  acquisition.  This  requires  incorporation  of  the  principle  in  policy 
statements,  leadership  actions,  and  contractual  provisions  and  incentives. 

2.  The  SecDef  should  establish  the  same  authority  for  the  USecAF  for  DOD  space 
programs  as  the  DNRO  has  for  implementing  the  National  Reconnaissance 
Program  (NRP)  budget. 

3.  To  ensure  realistic  budgets  and  cost  estimates,  the  USecAF/DNRO  should 

•  Direct  that  space  acquisition  programs  be  budgeted  to  a  most  probable 
(80/20)  cost,  with  a  20-25  percent  management  reserve  for  development 
programs  included  within  this  cost;  also  direct  that  reserves  are  not  to  be 
used  for  new  requirements; 

•  Direct  that  source  selections  evaluate  contractor  cost  credibility  and  use 
the  estimate  as  a  measure  of  their  technical  understanding; 

•  Conduct  more  effective  independent  cost  estimates  and  program 
assessments  and  incorporate  the  results  into  the  program  budget  and  plan; 

•  Implement  independent  senior  advisory  reviews  at  critical  acquisition 
milestones  with  experienced,  respected  outsiders. 

4.  The  USecAF/DNRO  should  compete  space  system  acquisitions  only  when 
clearly  in  the  best  interest  of  the  government  (e.g.,  new  mission  capability,  major 
new  technology,  or  poor  incumbent  perfonnance).  When  a  competition  occurs 
and  a  nonincumbent  is  the  winner,  the  loss  of  investment  in  the  losing  incumbent 
must  be  reflected  in  the  program  budget  and  plan.  In  addition,  provisions  must  be 
made  to  assure  continuity  between  the  legacy  system  and  the  new  system. 
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5.  SecDef  and  the  Director  of  Central  Intelligence  (DCI)  should  designate  senior 
leaders  in  the  DOD  and  intelligence  community  with  authority  to  lead  their 
respective  requirements  processes  for  national  security  space  systems.  The  senior 
leaders  must  have  the  support  necessary  to  assess — technically  and  fiscally — 
proposed  requirements  and  the  authority  to  couple  requirements  with  funding. 

6.  The  USecAF/DNRO  should  authorize  the  program  manager  to  control 
requirements  within  the  approved  baseline.  The  program  manager  should 
continuously  trade  and  challenge  requirements  throughout  the  program  life  cycle. 
Significant  requirements  changes  should  require  the  approval  of  the  senior  leaders 
for  requirements. 

7.  The  Commander,  Air  Force  Space  Command,  should  complete  the  ongoing 
effort  to  establish  a  dedicated  career  field  for  space  operations  and  acquisition 
personnel. 

8.  The  USecAF/DNRO  should  require  that  key  program  management  tours  be  a 
minimum  of  4  years. 

9.  The  USecAF/DNRO  should,  through  policy  and  leadership  action,  clearly 
define  the  responsibility,  authority,  and  accountability  for  program  managers, 
recognizing  the  criticality  of  program  managers  to  the  success  of  their  programs. 
In  selecting  managers,  acquisition  experience  must  be  a  prerequisite. 

10.  USecAF/DNRO  should  develop  a  robust  systems  engineering  capability  to 
support  program  initiation  and  development.  Specifically,  USecAF/DNRO  should 

•  Reestablish  organic  government  systems  engineering  capability  by 
selecting  appropriate  people  from  within  government,  hiring  to  acquire 
needed  capabilities,  and  implementing  training  programs;  and 

•  In  the  near  term,  ensure  full  utilization  of  the  combined  capabilities  of 
government,  Federally  Funded  Research  and  Development  Center 
(FFRDC),  and  systems  engineering  and  technical  assistance  (SETA) 
system  engineering  resources. 

11.  The  USecAF/DNRO  should  require  program  managers  to  identify  and  report 
potential  problems  early. 

•  Program  managers  should  establish  early  warning  metrics  and  report 
problems  up  the  management  chain  for  timely  corrective  action. 

•  Severe  and  prominent  penalties  should  follow  any  attempt  to  suppress 
problem  reporting. 
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12.  The  USecAF/DNRO  should  demand  that  national  security  space  contractors 

•  Account  for  the  quality  of  their  program  implementation  and  for  mission 
success, 

•  Identify  proven  management  and  engineering  practices  and  ensure  they 
are  being  utilized,  and 

•  Account  for  the  early  identification  and  open  discussion  of  problems  in 
their  program. 

13.  Program  managers  should  align  contract  and  fee  structure  to  focus  industry 
attention  on  proven  management  and  engineering  practices  and  mission  success. 
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Appendix  E  -  Excerpts  from  the  Systems  Engineering  Leading  Indicators  Guide 


Table  36.  Systems  Engineering  Leading  Indicators  Overview  (Massachusetts 
Institute  of  Technology,  INCOSE,  and  PSM,  2010) 


Leading 

Indicator 

Insight  Provided 

Phases  /  Stages 

P 

1 

P 

2 

P 

3 

P 

4 

P 

5 

S 

1 

s 

2 

S 

3 

S 

4 

S 

5 

Requirements 

Trends 

Rate  of  maturity  of  the  system  definition  against  the 
plan.  Additionally,  characterizes  the  stability  and 
completeness  of  the  system  requirements  that  could 
potentially  impact  design,  production,  operational  utility, 
or  support. 

System 

Definition 

Change  Backlog 
Trend 

Change  request  backlog  which,  when  excessive,  could 
have  adverse  impact  on  the  technical,  cost  and  schedule 
baselines. 

• 

• 

• 

• 

• 

• 

Interface 

Trends 

Interface  specification  closure  against  plan.  Lack  of 
timely  closure  could  pose  adverse  impact  to  system 
architecture,  design,  implementation  and/or  V&V  any  of 
which  could  pose  technical,  cost  and  schedule  impact. 

Requirements 

Validation 

Trends 

Progress  against  plan  in  assuring  that  the  customer 
requirements  are  valid  and  properly  understood. 

Adverse  trends  would  pose  impacts  to  system  design 
activity  with  corresponding  impacts  to  technical,  cost  & 
schedule  baselines  and  customer  satisfaction. 

Requirements 

Verification 

Trends 

Progress  against  plan  in  verifying  that  the  design  meets 
the  specified  requirements.  Adverse  trends  would 
indicate  inadequate  design  and  rework  that  could 
impact  technical,  cost  and  schedule  baselines.  Also, 
potential  adverse  operational  effectiveness  of  the 
system. 

Work  Product 
Approval 

Trends 

Adequacy  of  internal  processes  for  the  work  being 
performed  and  also  the  adequacy  of  the  document 
review  process,  both  internal  and  external  to  the 
organization.  High  reject  count  would  suggest  poor 
quality  work  or  a  poor  document  review  process  each  of 
which  could  have  adverse  cost,  schedule  and  customer 
satisfaction  impact. 

Review  Action 
Closure  Trends 

Responsiveness  of  the  organization  in  closing  post¬ 
review  actions.  Adverse  trends  could  forecast  potential 
technical,  cost  and  schedule  baseline  issues. 

Risk  Exposure 
Trends 

Effectiveness  of  risk  management  process  in  managing  / 
mitigating  technical,  cost  &  schedule  risks.  An  effective 
risk  handing  process  will  lower  risk  exposure  trends. 

Risk  Treatment 
Trends 

Effectiveness  of  the  SE  organization  in  implementing 
risk  mitigation  activities.  If  the  SE  organization  is  not 
retiring  risk  in  a  timely  manner,  additional  resources  can 
be  allocated  before  additional  problems  are  created. 

Technology 
Maturity  Trends 

Risk  associated  with  incorporation  of  new  technology  or 
failure  to  refresh  dated  technology.  Adoption  of 
immature  technology  could  introduce  significant  risk 
during  development  while  failure  to  refresh  dates 
technology  could  have  operational 
effectiveness/customer  satisfaction  impact. 

• 

• 

• 

• 

• 

• 

• 

Technical 

Measurement 

Trends 

Progress  towards  meeting  the  Measures  of  Effectiveness 
(MOEs)  /  Performance  (MOPs)  /  Key  Performance 
Parameters  (KPPs)  and  Technical  Performance  Measures 
(TPMs).  Lack  of  timely  closure  is  an  indicator  of 
performance  deficiencies  in  the  product  design  and/or 
project  team's  performance. 

• 

• 

Systems 
Engineering 
Staffing  &  Skills 
Trends 

Quantity  and  quality  of  SE  personnel  assigned,  the  skill 
and  seniority  mix,  and  the  time  phasing  of  their 
application  throughout  the  project  lifecycle. 
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Leading 

Insight  Provided 

Phases  /  Stages 

Indicator 

P 

P 

P 

P 

P 

S 

s 

S 

S 

S 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

Process 

Compliance 

Trends 

Quality  and  consistency  of  the  project  defined  SE 
process  as  documented  in  SEP/SEMP.  Poor/inconsistent 

SE  processes  and/or  failure  to  adhere  to  SEP/SEMP, 
increase  protect  risk. 

Facility  and 
Equipment 
Availability 
Trends 

Availability  of  non-personnel  resources  (infrastructure, 
capital  assets,  etc.)  needed  throughout  the  project 
lifecycle. 

Defect/ Error 
Trends 

Progress  towards  the  creation  of  a  product  or  the 
delivery  of  a  service  that  meets  the  quality  expectations 
of  its  recipient.  Understanding  the  proportion  of  defects 
being  found  and  opportunities  for  finding  defects  at 
each  stage  of  the  development  process  of  a  product  or 
the  execution  of  a  service. 

System 

Affordability 

Trends 

Progress  towards  a  system  that  is  affordable  for  the 
stakeholders.  Understanding  the  balance  between 
performance,  cost,  and  schedule  and  the  associated 
confidence  or  risk. 

Architecture 

Trends 

Maturity  of  an  organization  with  regards  to 
implementation  and  deployment  of  an  architecture 
process  that  is  based  on  an  accept  set  of  industry 
standards  and  guidelines. 

• 

• 

• 

• 

• 

Schedule  and 
Cost  Pressure 

Impact  of  schedule  and  cost  challenges  on  carrying  out 
a  project 
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Table  37.  Leading  Indicator  Specification  Example  (Massachusetts  Institute  of 


Technology,  INCOSE,  and  PSM,  2010) 


3.1.1  Requirements  Trend  Specification 


Requirements  Trends 

Information  Need  Description 

Information 

Need 

•  Evaluate  the  stability  and  adequacy  of  the  requirements  to  understand 
the  risks  to  other  activities  towards  providing  required  capability,  on- 
time  and  within  budget. 

•  Understand  the  growth,  change,  completeness  and  correctness  of  the 
definition  of  the  system  requirements. 

Information 

Category 

1.  Product  size  and  stability  -  Functional  Size  and  Stability 

2.  Also  may  relate  to  Product  Quality  and  Process  Performance  (relative  to 
effectiveness  and  efficiency  of  validation) 

Measurable  Concept  and  Leading  Insight 

Measurable 

Concept 

Is  the  SE  effort  driving  towards  stability  in  the  System  definition  and  size? 

Leading  Insight 
Provided 

•  Indicates  whether  the  system  definition  is  maturing  as  expected. 

•  Indicates  risks  of  change  to  and  quality  of  architecture,  design, 
implementation,  verification,  and  validation. 

•  Indicates  schedule  and  cost  risks. 

•  Greater  requirements  growth,  changes,  or  impacts  than  planned  or 
lower  closure  rate  of  TBDs/TBRs  than  planned  indicate  these  risks. 

•  May  indicate  future  need  for  different  level  or  type  of  resources/skills. 

•  Indicates  potential  lack  of  understanding  of  stakeholder  requirements 
that  may  lead  to  operational  or  supportability  deficiencies. 

Base  Measure  Specification 

Base  Measures 

1.  Requirements 

2.  Requirement  TBDs/TBRs 

3.  Requirement  Defects 

4.  Requirement  Changes 

5.  Requirement  Change  Impact 

Measurement 

Methods 

1.  Count  the  number  of  Requirements  (record  associated  attributes  of 
interest;  e.g.,  Process  Phases,  Disposition  Action,  Maturity  States, 

Priority  Levels,  Cause,  Impact  Level,  Classification  Type,  and  Dates  & 
Times) 

2.  Count  the  number  of  Requirement  TBDs/TBRs  (record  associated 
attributes  of  interest;  e.g.,  Process  Phases,  Disposition  Action,  Maturity 
States,  Priority  Levels,  Cause,  Impact  Level,  Classification  Type,  and 

Dates  &  Times) 

3.  Count  the  number  of  Requirement  Defects  (record  associated  attributes 
of  interest;  e.g.,  Process  Phases,  Disposition  Action,  Maturity  States, 
Priority  Levels,  Cause,  Impact  Level,  Classification  Type,  and  Dates  & 
Times) 

4.  Count  the  number  of  Requirement  Changes  (record  associated  attributes 
of  interest;  e.g.,  Process  Phases,  Disposition  Action,  Maturity  States, 
Priority  Levels,  Cause,  Impact  Level,  Classification  Type,  and  Dates  & 
Times) 

5.  Estimate  the  impact  of  a  Requirement  Change 
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Requirements  Trends 

Unit  of 
Measurement 

1.  Requirements 

2.  Requirement  TBDs/TBRs  per  associated  attributes 

3.  Requirement  Defects  per  associated  attributes 

4.  Requirement  Changes  per  associated  attributes 

5.  Effort  Hours  per  Requirement  Change  (effort  hours  or  range  of  effort 
hours  expected  for  each  change) 

Entities  and  Attributes 

Relevant  Entities 

•  Requirements 

Attributes 

•  Requirement  TBDs/TBRs 

•  Requirement  Defects 

•  Requirement  Changes 

•  Additional  attributes  including  but  not  limited  to  the  Process  Phases, 
Disposition  Action,  Maturity  States,  Priority  Levels,  Cause,  Impact  Level, 
Classification  Type,  and  Dates  &  Times  coupled  with  the  associated 
events 

Derived  Measure  Specification 

Derived  Measure 

1.  %  Requirements  Approved 

2.  %  Requirements  Growth 

3.  %  TBDs/TBRs  Closure  Variance  per  Plan 

4.  %  Requirements  Modified 

5.  Estimated  Impact  of  Requirements  Changes  for  a  given  time  interval  (in 
Effort  Hours) 

6.  Requirement  Defect  Profile 

7.  Requirement  Defect  Density 

8.  Requirement  Defect  Leakage  (or  Escapes) 

9.  Cycle  time  for  Requirement  Chanqes  (each  and  averaqe) 

Measurement 
Function  * 

1.  (Requirements  Approved  /  Requirements  identified  and  defined)*  100  for 
a  given  time  interval 

2.  ((Requirements  in  current  baseline  -  Requirements  in  previous  baseline) 

/  (Requirements  in  previous  baseline)  *  100 

3.  ((TBDs/TBRs  planned  for  closure  -  TBDs/TBRs  closed)  /  TBDs/TBRs 
planned  for  closure)  *  100 

4.  (Requirements  Modified  /  Total  Requirements)  *  100  for  a  given  time 
interval 

5.  Sum  of  estimated  impacts  of  Requirement  Changes  during  a  given  time 
interval 

6.  Requirement  Defects  for  each  defect  category 

7.  Requirement  Defects  /  Requirements  as  a  function  of  time 

8.  Subset  of  Requirement  Defects  found  in  a  phase  subsequent  to  its 
insertion 

9.  Elapsed  time  (difference  between  start  and  stop  times)  or  total  effort 
hours  for  each  Requirements  Change 
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Requirements  Trends 

Indicator  Specification 

Indicator 
Description  and 
Sample 

Line  or  bar  graphs  that  show  trends  of  requirements  growth  and  TBD/TBR 
closure  per  plan.  Stacked  bar  graph  that  shows  types,  causes,  and 
impact/severity  of  changes.  Show  thresholds  of  expected  values  based  on 
experiential  data.  Show  key  events  along  the  time  axis  of  the  graphs. 

1.  Line  or  bar  graphs  that  show  growth  of  Requirements  over  time 

2.  Line  or  bar  graphs  that  show  %  Requirements  Approved  over  time 

3.  Line  or  bar  graphs  that  show  %  TBDs/TBRs  not  closed  per  plan 

4.  Line  or  bar  graphs  that  show  %  Requirements  Change 

5.  Line  or  bar  graphs  that  show  Estimated  Impact  of  Requirements  Change 

for  a  given  time  interval  (in  effort  hours) 

6.  Line  or  bar  graphs  that  show  Defect  Profile  (by  types,  causes,  severity, 
etc.) 

7.  Line  or  bar  graphs  that  show  Defect  Density 

8.  Stacked  bar  graph  that  shows  types,  causes,  and  impact/severity  of 
Requirements  Chanqes 

Thresholds  and 
Outliers 

Organization  dependent. 

Decision  Criteria 

Investigate,  and  potentially,  take  corrective  action  when  the  requirements 
growth,  requirements  change  impact,  or  defect  density/distribution  exceeds 
established  thresholds  <fill  in  organization  specific  threshold>  or  a  trend  is 
observed  per  established  guidelines  <fill  in  organizational  specific:*. 

Indicator 

Interpretation 

•  Used  to  understand  the  maturity  of  the  system  definition 

•  Used  to  understand  impact  on  system  definition  and  impact  on 
production. 

•  Analyze  this  indicator  for  process  performance  and  other  relationships 
that  may  provide  more  "leading  perspective". 

•  Ops  Concept  quality  may  be  a  significant  leading  indicator  of  the 
requirements  stability  (may  be  able  to  use  number  of  review 
comments;  stakeholder  coverage  in  defining  the  Ops  Concept). 

•  Care  should  be  taken  that  the  organization  does  not  create  incentives 
driving  perceptions  that  all  requirements  change  is  undesirable.  Note: 
Requirements  changes  may  be  necessary  to  accommodate  new 
functionality. 

•  Review  of  this  indicator  can  help  determine  the  adequacy  of: 

o  Quantity  and  quality  of  Systems  Engineers 
o  Infrastructure 

o  Process  maturity  (acquirer  and  supplier) 

o  Interface  design  capability 
o  Stakeholder  collaboration  across  life  cycle 

•  Funding  by  customer;  financial  challenge  by  the  program  management 
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Requirements  Trends 

Additional  Information 

Related 

Processes 

Stakeholder  Requirements,  Requirements  Analysis,  Architectural  Design 

Assumptions 

•  Requirements  Database,  Change  Control  records,  defect  records  are 
maintained  &  current. 

•  TBDs  and  TBRs  are  recorded  and  tracked. 

Additional 

Analysis 

Guidance 

•  May  also  be  helpful  to  track  trends  based  on  severity/priority  of  changes 

•  Defect  leakage  -  identify  the  phases  in  which  defect  was  inserted  and 
found  for  each  defect  recorded. 

Implementation 

Considerations 

•  Requirements  that  are  not  at  least  at  the  point  of  a  draft  baseline  should 
not  be  counted. 

•  Usage  is  driven  by  the  correctness  and  stability  of  requirements 
definition. 

o  Lower  stability  means  higher  risk  of  impact  to  other  activities 
and  other  phases,  thus  requiring  more  frequent  review, 
o  Applies  throughout  the  life  cycle,  based  on  risk, 
o  Track  this  information  per  baseline  version  to  track  the  maturity 
of  the  baseline  as  the  system  definition  evolves. 

User  of 
Information 

•  Program/Project  Manager 

•  Chief  Systems  Engineer 

•  Product  Managers 

•  Designers 

Data  Collection 
Procedure 

•  See  Appendix  F 

Data  Analysis 
Procedure 

•  See  Appendix  F 
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Appendix  F  -  SAIMM  Interoperability  Measurement  Checklist 


Step  1  -  Determine  Program  Interoperation  Maturity 

Assess  the  subject  program1  s  interoperation  maturity  for  the  cost,  schedule  and 
requirements  areas  using  the  SAIMM  matrix  provided  in  Chapter  III  (SAIMM  link).  The 
program  may  tailor  the  matrix  based  upon  their  historical  precedent  and  lessons  learned. 

Step  2  -  Instantiate  the  Program  Character  States 

Place  the  respective  SAIMM  scores  for  each  area  and  attribute  into  a  spreadsheet 
or  similar  tool.  The  following  format  was  used  to  perform  the  calculations  in  this  thesis: 


Table  38.  Instantiation  Example  Format 


Program  X 

Mission  Focus 

Stability 

Discipline 

Realism 

Cost 

1 

1 

1 

1 

Schedule 

1 

1 

1 

1 

Requirements 

1 

1 

1 

1 

Step  3  -  Perform  the  Interoperability  Measurement 

Apply  the  SiniReai  function  to  the  instantiation  matrix  to  perfonn  the 
interoperability  measurement.  Set  r  to  2,  and  n  and  cmax  to  4. 

Step  4  -  Evaluate  the  Results 

Examine  the  resulting  interoperability  scores.  The  program  may  use  the  AEHF, 
NPOESS  and  SBIRS  examples  in  this  thesis  for  comparison.  The  program  may  also 
perfonn  the  measurement  during  various  phases  in  the  program  to  evaluate  progress. 
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